The officially official Devuan Forum!

nahkhiirmees

Member

Registered: 2022-07-24

Posts: 261  

sha512sum...

When i do this:

cp a/*.xx b
cp a/*.yy b
cp a/*.zz b
cd b
sha512sum ./* > sums.txt
sha512sum -c sums.txt

i get OK's from all of the files.
on the other hand when i do this:

cd a
sha512sum ./*.xx > part1.txt
sha512sum ./*.yy >> part1.txt
sha512sum ./*.zz > part2.txt
cp a/*.xx b;cp a/*.yy b;cp a.zz b;cp part*.txt b
cd b
mv part1.txt sums.txt
cat part2.txt >> sums.txt
sha512sum -c sums.txt

i get complaints from lines containing .yy and .zz files.
Have you encountered this kind of behaviour from sha512sum?

Trying to find out if it's either:
a) you're not supposed to use sha512sum(and other commands) like that (the second code-block)
b) i have found a bug in sha512sum
c) there's something wrong with my current desktop

i bought the hard drive abt year ago. And did a 24 hour memtest before installing OS. Passed with flying colours.
So i would like to think it is either a) or b) .

Last edited by nahkhiirmees (2023-02-16 23:00:02)

ralph.ronnquist

Administrator

From: Battery Point, Tasmania, AUS

Registered: 2016-11-30

Posts: 1,254  

Re: sha512sum...

Are those the actual commands?
Are the files in a/athe same as in a/?
Or have you transcribed into some notional commands to illustrate what you wanted to do?
(or actually a near miss thereof)

I don't think any of your options applies, since it very likely is just a user error.

Camtaf

Member

Registered: 2019-11-19

Posts: 436  

Re: sha512sum...

You seem to be replacing the content each time you 'cp' to the file 'b'.

cp a/*.xx b;cp a/*.yy b;cp a.zz b;cp part*.txt b

nahkhiirmees

Member

Registered: 2022-07-24

Posts: 261  

Re: sha512sum...

Last night, while trying to distillate the essential parts of the problem, it seems that i was not precise enough. Noise is needed along with the signal.
Another try:

Try 1

mount /dev/sdaX /mnt/a -o ro
mount /dev/sdaY /mnt/b -o rw

cp /mnt/a/*.xx /mnt/b
cp /mnt/a/*.yy /mnt/b
cp /mnt/a/*.zz /mnt/b
cd /mnt/b
sha512sum ./* > sums.txt
sha512sum -c sums.txt

sha512sum says the files in the list are all OK
after that another round:

umount /mnt/a
rm -rf /mnt/b/*
mount /dev/sdaX /mnt/a -o ro
cd /mnt/a
sha512sum ./*.xx > part1.txt
sha512sum ./*.yy >> part1.txt
sha512sum ./*.zz > part2.txt
cp /mnt/a/*.xx /mnt/b;cp /mnt/a/*.yy /mnt/b;cp /mnt/a/*.zz /mnt/b;cp /mnt/a/part*.txt /mnt/b
cd /mnt/b
mv part1.txt sums.txt
cat part2.txt >> sums.txt
sha512sum -c sums.txt

and only .xx files receive OK from sha512sum -c .
.zz files are supposed to change more often than .yy files and .yy files are supposed to change more often than .xx - files. So it seemed like a good idea to split the file sums.txt in 3 parts. Or to be more precise, build the file sums.txt from 2 or 3 files.

I tried to look at those .txt files with hexdump. I may have found the characters that cause the problem with "sha512sum -c" but i haven't figured out how to get completely rid of them.

Last edited by nahkhiirmees (2023-02-17 14:05:09)

nahkhiirmees

Member

Registered: 2022-07-24

Posts: 261  

Re: sha512sum...

Should it be possible, to combine a working digest list from smaller ones, with cat, mv and sha512sum?

chris2be8

Member

Registered: 2018-08-11

Posts: 307  

Re: sha512sum...

The first thing I'd try is to repeat round 1, cp sums.txt sums_1.txt, repeat round 2 and diff sums_1.txt sums.txt to see if they differ.

nahkhiirmees

Member

Registered: 2022-07-24

Posts: 261  

Re: sha512sum...

Should it be possible, to combine a working digest list from smaller ones, with cat, mv and sha512sum?

I'll answer to myself:it is possible.

nahkhiirmees

Member

Registered: 2022-07-24

Posts: 261  

Re: sha512sum...

Are those the actual commands?
Are the files in a/athe same as in a/?
Or have you transcribed into some notional commands to illustrate what you wanted to do?
(or actually a near miss thereof)

I don't think any of your options applies, since it very likely is just a user error.

The actual commands are a bit different from the previous posts. I thought i could reproduce the problem that way but no.

The basic idea is:"copy files mentioned in this list, from source to destination and make also a list of digests containing almost all of the mentioned files".

for file in $list do
  cp $sourcedir/$file $destdir
  tmp=$(sha512sum $sourcedir/$file | grep -v $not_this_file)
  echo $tmp >> part1.txt
done

sha512sum $sourcedir/*.zz > part2.txt
cp $sourcedir/*.zz $destdir
cp part*.txt $destdir

cd $destdir
mv part1,txt sums.txt
cat part2.txt >> sums.txt

would be a very close approximation to the actual script. I guess it's the second or third row inside for-loop where things go wrong. Haven't found yet a perfect solution. With awk, i got rid of most of the complaints from "sha512sum -c".

Last edited by nahkhiirmees (2023-02-17 19:35:21)

ralph.ronnquist

Administrator

From: Battery Point, Tasmania, AUS

Registered: 2016-11-30

Posts: 1,254  

Re: sha512sum...

tmp=$(sha512sum $sourcedir/$file | grep -v $not_this_file)

sha512sum $sourcedir/*.zz > part2.txt

Have you looked at what those "sha512sum" runs generate?

Yes, you may concatenate "shasum.txt" snippets. But you must be sure that the pathnames are the correct ones for when that "shasum.txt" is used.

Afaict, in your example (in the previous post) the pathnames in the file include "$sourcedir/" as path prefix, and therefore the checksum verification needs to be done with the files being in the "$sourcedir/" subdirectory of the current working directory.

If I may venture an opinion though, it generally makes the code less complicated and more easily reviewable when having a "shasum.txt" for all files in a directory be generated only once, when that directory is fully populated. Basically, your examples "like this, it works " are examples of good coding practice whereas the other examples "like this, it doesn't work" leave a bit in that respect. Which perhaps illustrates the point.

nahkhiirmees

Member

Registered: 2022-07-24

Posts: 261  

Re: sha512sum...

I have taken account the path. I think the problem are some non-visible characters in the lists. Making the hash list in many parts seemed like a good idea for a while but now i'm not so sure anymore. Maybe i'll upload the full file or partials into pastebin some day.

nahkhiirmees

Member

Registered: 2022-07-24

Posts: 261  

Re: sha512sum...

I looked at those pre-mentioned scripts again. And then remembered that first i commented out the rows that generated the partial digests. And after that, removed most of the commented lines. So now it is bit difficult to reproduce the problem-causing files.
Anyways, i found a partial digest file and uploaded it to pastebin.com . It looks a bit weird(i mean those empty lines, not filenames, those are obfuscated) but "sha512sum -c" didn't complain. pastebin.com/4aCjnCLe
I can show which commands i used to make that file. For...do...done, awk, echo and sha51sum were used. That much i can tell.
Mostly pointless, i know.

Last edited by nahkhiirmees (2023-05-04 18:11:45)

nahkhiirmees

Member

Registered: 2022-07-24

Posts: 261  

Re: sha512sum...

I have been planning to reinstall os to my desktop so when i get that done, i hope that all the weird behaviour goes away or i find out if my hardware is faulty.