[Discuss] rsync v. cp in data migration

[gaf at blu.org (Jerry Feldman)]

On 05/26/2013 10:58 AM, Derek Martin wrote:
> On Sun, May 26, 2013 at 07:36:14AM -0400, Jerry Feldman wrote:
>>> Generally the convention is to use upper case names for environment
>>> variables (i.e. things you want to have persistence across
>>> sessions/processes/etc.), and shell scripts should use lower case
>>> names for their variables to avoid unintended conflicts like this.
>> Yes, using upper case for environment variables and lower case for
>> local variables is the long-time Unix/Linux convention. Shell
>> scripts can use both local as well as environment variables. I think
>> your definition of persistence is incorrect.
>> Environment variables persist within the process where they were
>> created, and any child process.
> I never actually gave you my definition of persistence, and I've been
> posting here long enough that I think you know that I know how
> environment variables work. I was vague for the sake of brevity, but
> being vague is not the same as being wrong. ;-)
>
> It's perhaps interesting to note that if you do change an environment
> variable in your shell, but it is one that is set in your login files,
> then when you start a new shell from that one, the changed value will
> "revert" to the value it had when you log in, on account of it having
> been re-read from your login files (or rather it MAY, whether or not
> this happens depends upon how the new shell was invoked, as described
> in the INVOCATION section of the bash man page).
>
> And since you bring it up, and it's tangentially related to Greg's
> script, I'll mention that when you pipe the output of a loop in a
> script, as I did in the example that I gave, that loop is run in a
> subshell.  This is important to know because it means that when you do
> this, if, inside that loop, you change a variable that exists outside
> the loop, its value will revert to the value it had before the start
> of the loop.  For example, take the following script:
>
> $ cat x.sh
> #!/bin/sh
>
> x=hello
> for i in 1 2 3; do
>      x="$x$i"
>      echo $x
> done
> echo $x
>
> When you run this, you get the output you would intutively expect:
>
> $ sh x.sh
> hello1
> hello12
> hello123
> hello123
>
> But if we make a small modification:
>
> $ cat y.sh
> #!/bin/sh
>
> x=hello
> for i in 1 2 3; do
>      x="$x$i"
>      echo $x
> done | tee /dev/null
> echo $x
>
> Your output becomes:
>
> $ sh y.sh
> hello1
> hello12
> hello123
> hello
>
>
> This behavior catches a lot of people who shell script casually
> unawares.  Shell scripting is a virtual mine field...  I used to be a
> big fan of it, but these days I'll prefer Python for almost
> everything, save really short scripts that mostly just run a few
> commands, or the occasional very short Perl script that does very
> little other than process some regexes.
>
>
I have considered using Python and I consider Python to be excellent. At 
work, I have been using tcl./tk. The problem with Python is that Python 
3 is not backwards compatible to scripts written in Python 2+. The 
difference between x.sh and y.sh is in y.sh the entire iteration, 
for..done is actually spun off as a separate process, so that the 
original value of x is not replaced. So, if you add a ps after the echo 
within the iteration you will see that you have 2 y.sh processes 
running. This is because of the pipeto tee.
IMHO, shell scripts should not be used essentially as application 
programs. Another reason is speed. We had an application to convert 
mailman passwords to web passwords written in shell script. The 
application took something like 30 minutes or even longer. I rewrote the 
application in C++, and cut it down to about 10 mins. I then rewote it 
in Python and it runs in seconds. (Part of the reason is that Mailman is 
written in Python and you can access some of the stuff directly rather 
than spawning a process).

-- 
Jerry Feldman <gaf at blu.org>
Boston Linux and Unix
PGP key id:3BC1EB90
PGP Key fingerprint: 49E2 C52A FC5A A31F 8D66  C0AF 7CEA 30FC 3BC1 EB90