8 processors but still waiting

Our new build system has lots of ram and a multicore hyperthreaded cpu: 4*2 processors. It is just unbelievable how responsive this Vista-64 machine is.

We are currently making some changes to the build process of WatchFTP, creating separate installers for each supported language. Previously, we created just one installer with a multilingual executable (thank you Sisulizer for a great localization product). The multilingual approach worked fine most of the time, however when a German customer living in France wants to run the English version of WatchFTP on his Spanish-language operating system it was just not possible. The operating system would select the language to display, not much you could do about it.

Future WatchFTP versions will have the desired display language build-in. Sisulizer will create language specific (German, Spanish etc) copies of the original (English) executables. The display language will only depend on the installer you download from our websites.

Steps in our build process

1. Build a "release" version of WatchFTP
   (Microsoft Visual Studio 2008 - C++ by the way)
2. Run Sisulizer to create the multilingual executables
3. Protect the core components (2 executables)
4. Code-sign the EXE and DLLs
5. Run Inno Setup to create the installer
6. Code-sign the setup file
7. Test the setup
   When we find issues, fix them and go back to step 1.
8. When all is OK, a lot of other tasks like uploading to our website etc

The most time consuming step (several minutes) was step 3 (software protection). Not really a big deal, we don't release new versions that often. However, step 7 (test setup) can cause a lot of rebuilds (start at step 1) and the delays for software protection become very annoying.

It gets Worse!!

When we implement separate installers for the different languages we support, we will have to do step 3 (protect) for each language (currently German, Spanish and English. Soon we will have a French version as well).
This will make our already long build process 4 times longer - totally unacceptable.

Our great new computer doesn't help at all because all those steps need to be sequential. We can not code-sign (step 4) before the protection (step 3) is completed. Also, the protection phase is done with a single-threaded program, even if the computer has 1000 cpus, it will only use one of them.

How we solved it

Obviously, the protection step (nr 4) needs to be paralleled. We can launch the protector as a separate process for each of the languages. A first attempt (WRONG!!) in our build batch script:

....
ECHO ==STEP 2 Complete==>>C:\Temp\buildlog.txt
ECHO ==STEP 3 Starts - Software Protection==>>C:\Temp\buildlog.txt
Set LANGUAGE=ENGLISH
Start "Protecting %LANGUAGE%" "E:\Build\Protect.bat"
Set LANGUAGE=SPANISH
Start "Protecting %LANGUAGE%" "E:\Build\Protect.bat"
Set LANGUAGE=GERMAN
Start "Protecting %LANGUAGE%" "E:\Build\Protect.bat"
Set LANGUAGE=FRENCH
Start "Protecting %LANGUAGE%" "E:\Build\Protect.bat"
ECHO ==STEP 3 Complete==>>C:\Temp\buildlog.txt
ECHO ==STEP 4 Starts - Code Signing protected Executables ==>>C:\Temp\buildlog.txt

(note: the scripts shown in this post are very simplified without error checking etcetera).
The protect.bat file is called and based on the LANGUAGE variable it will protect a specific language version of WatchFTP. The beauty of the Start command is that it will launch a separate console (command prompt) to run the protect.bat script. In the example above 4 language versions are protected simultaneously, each using their own CPU. Obviously, because of disk usage, there is some slowdown - it won't complete 4 times as fast.

A small problem...

In case you didn't realize, the above script is sort of wrong: it continues running while the 4 "Protect.bat" files are not finished yet. So it gladly goes on to Step 4 (code signing), but there isn't anything to code-sign yet. We need to wait for the 4 protect.bat processes to complete before we can continue with Step 4.

And an easy fix

We have a small command prompt program called GdPUtil (Free Download). Please read the description of the -join parameter.

We added the following lines at the end of protect.bat so it signals to the main script it is ready:

....
"C:\Bin\GdPUtil.exe" -join protect%LANGUAGE% 1000
EXIT

And changed the original build script to

....
ECHO ==STEP 2 Complete==>>C:\Temp\buildlog.txt
ECHO ==STEP 3 Starts - Software Protection==>>C:\Temp\buildlog.txt
Set LANGUAGE=ENGLISH
Start "Protecting %LANGUAGE%" "E:\Build\Protect.bat"
Set LANGUAGE=SPANISH
Start "Protecting %LANGUAGE%" "E:\Build\Protect.bat"
Set LANGUAGE=GERMAN
Start "Protecting %LANGUAGE%" "E:\Build\Protect.bat"
Set LANGUAGE=FRENCH
Start "Protecting %LANGUAGE%" "E:\Build\Protect.bat"

ECHO ==STEP 3 waiting for protection to complete ==>>C:\Temp\buildlog.txt

"C:\Bin\GdPUtil.exe" -join protectENGLISH 1000
"C:\Bin\GdPUtil.exe" -join protectSPANISH 1000
"C:\Bin\GdPUtil.exe" -join protectGERMAN 1000
"C:\Bin\GdPUtil.exe" -join protectFRENCH 1000

ECHO ==STEP 3 Complete==>>C:\Temp\buildlog.txt
ECHO ==STEP 4 Starts - Code Signing protected Executables ==>>C:\Temp\buildlog.txt

Now our script works like a champ! Even though it has to do 4 times as much work, it is only slightly slower than it used to be.

Why does this matter to you?

If you have long running scripts, have a good look if they can be broken up like we did in our build script. For example, are you uploading a file to multiple FTP servers in your script? If you have enough upload bandwidth but it is the FTP servers causing the slowdown, this technique can be very helpful.

SkunkWorks - New Program, New Name, Contest, Testers.....

Hello,

We are currently working on a new program. Functionality is going great so far but we have a strange problem ;-) More on that further down below...

The Program

So far our programs WatchDirectory and WatchFTP concentrated on doing things automagically. This new program is different.
It allows you to select files (in Windows Explorer, for example), add these files to a "basket" (an icon floating above explorer) and then perform "actions" on those baskets. A few actions we are currently implementing:

• COPY - copy the files dropped into the basked to a pre-configured directory.
• MOVE - same as copy, but move the files instead.
• EMAIL - email the files to a pre-configured email address.
• FTP - upload the files to an FTP server.
• BATCH (you already knew this was coming, I guess ;-) - Run a script for each of those files - program your own action.
• YourIdea - we are always interested to hear from you.

The Name

So... if you read the above... any idea for a good name? There are a few important features the name must have:

• English - our biggest market.
• Web - the www.TheNameYouThinkOf.com name must be available.
• Attractive/Mysterious - if the name wows people who see it, making them curious, that would be great!
• Not offensive - need I explain this?
• File - it would be great if it has the word "file" in its name.
• Short - not as important as the above, but shorter names are easier to type (url in webbrowsers).

We already thought of several names (all very boring), do you know a good name? A good program name can make the difference between success or failure for a new program, so if you suggest the "winning name" to us we will compensate you for it. Beside a free copy of the program and our everlasting gratitude - a $1000 reward or 50% of the first month sales of this program - you choose!

Send an email to gert@watchdirectory.net, use the subject "SKUNKWORKS - Name". Tell us the program name and why you think it is a great name to use.

Beta Testers

It may take a month, maybe 2 months before we have the first beta ready, but we need beta testers for this new program. Beta testers receive a free copy when it is released, obviously. Interested?

Send an email to gert@watchdirectory.net, use the subject "SKUNKWORKS - Beta".

Update - July 22, 2009

We just bought the domain www.FileHalla.com (thank you Peter Wester) - any Vikings reading this ;-)
If you have a better idea, the contest is still open, probably until the end of the month.

Update - July 25, 2009

We just bought the domain www.FileViking.com (thank you Al Harberg). He didn't actually suggest this name but warned me about negative feelings the name Valhalla/Walhalla might cause.

Am I sure about FileViking? I think some ancestors of mine were slaughtered by Vikings... It has been some time ago, though...

Tweet your files with curl

This post shows how you can let WatchDirectory or WatchFTP post a new twitter message when they detect new files. Excuse me if I get some of the twitter terminology wrong - I grew up when computers were something you see in movies ;-)

Curl

This page shows that posting a message to twitter is as simple as

curl -u username:password -d status="your message here" http://twitter.com/statuses/update.json

So, we need Curl to post. I used the non-ssl version while testing. Make sure to download a Windows version from their download page.

Batch File

If you want to "tweet" when a new file is detected, create a new task and select the Run a Batch Script task (for WatchDirectory) or Run a Batch Script Action (for WatchFTP).

Let it run the following batch file (may not show correctly in all blog-readers):

@echo off
SET CURL=E:\bin\curl-7.19.5\curl.exe
SET TWUSER=TheTwitterUser
SET TWPASS=TheTwitterPassword

rem if we are called by WatchFTP, copy its variables to WD variables
rem so this script is compatible with both programs

IF "%WF_REASON%" NEQ "" (
  SET WD_REASON=%WF_REASON%
  SET WD_FILE=%WF_FILE%
)


GOTO :%WD_REASON%

:FILENEW
SET MSG="New file: %WD_FILE%"
GOTO :SendIt
:FILEDEL
SET MSG="Deleted file: %WD_FILE%"
GOTO :SendIt
:FILECHNG
SET MSG="Changed file: %WD_FILE%"
GOTO :SendIt
:FILEREN
SET MSG="Renamed file: %WD_OFILE% to %WD_FILE%"
GOTO :SendIt
:DIRNEW
SET MSG="New directory: %WD_FILE%"
GOTO :SendIt
:DIRDEL
SET MSG="Deleted directory: %WD_FILE%"
GOTO :SendIt
:DIRREN
SET MSG="Renamed directory: %WD_OFILE% to %WD_FILE%"
GOTO :SendIt


:SendIt
"%CURL%" -u %TWUSER%:%TWPASS% -d status=%MSG% http://twitter.com/statuses/update.json

Make sure to properly change the first few lines (where did you install curl, what is your twitter userid and password).

Twitter Limits

A tweet is a short message (I believe limited to about 140 characters). If your directory and filenames are long, you may need to shorten the messages a bit. For example, change %WD_FILE% to %WD_FILE_N% everywhere in the script so it will only tweet the filename (without the directory path).

Twitter also has some other limits you may hit if you call it too often.

Blogging your Files?

This post shows you how.

WatchFTP 2.2 beta

A beta version for WF 2.2 is available on our forum.

New in this beta release is that you can enter all kind of "timing" related settings for each task. Among other things, this gives you easier access to all the "secret settings" described in this blogpost.

This beta does not have the new features translated to German and Spanish yet - the new items will show in English.

To install (and keep your current settings and tasks):
* Do NOT uninstall your current WF version
* Install the beta and use all proposed settings of the installer (same directories)

WatchDirectory 4.5.9 released

We just released a new WatchDirectory version - another free upgrade for all WatchDirectory version 4 customers.

Changes in this release

• Fixed: The tip window sometimes caused the WD Control Center to crash at startup
• New: The Auto Copy plugin can now MOVE files. When you move files on the same disk, this is faster than copy followed by delete.
• New: The Folder Size plugin writes a timestamp and filename of deleted/moved files to <Task Directory>\<Task Name>\removedfiles_YYYYMMDD.csv. These files are automatically removed when they are a month old.
• New: The Folder Size plugin can check for several new "events"
• Fixed: The "Last Message" column of the WatchDirectory Control Center was not always updating properly
• Fixed: The Auto UNZIP task now supports far larger files (tested with a 2GB zip containing an 8GB file)
• Fixed: The Auto ZIP task now supports far larger files (tested up to 8GB)
• Fixed: The Auto Copy option to remove directories when they become empty in some cases also removed the monitored directory.
• New: The thumbnail plugin can now be used as a SubTask in the SubTask plugin
• New: The thumbnail plugin has a new option to prevent overwriting existing thumbnails
• New: You can set the "color" of a task. If you have a lot of tasks, this allows you to organize them in different "groups". Give a task one of 6 colors using the "right click" context menu. You can sort the tasks by color by clicking the header in the WD Control Center. You can select all tasks having the same color using the new "Colors Toolbar".
• New: When you let the email plugin combine emails, you can now set the separator between emails yourself.
• New: Use CTRL+F (or View -> Find) to find and select tasks.
• Fix: The Automatically Start any Program plugin did not properly set the environment variable WD_REASON when a new directory was detected.

How to install

Stop all your WD tasks, stop the WD Control Center. Now install the new version in the same location as your previous installed version (overwrite it).
This will keep all your settings and tasks intact.

Download

Download Here (English) or Here (German).

July Special - Free ShowSize license

When you buy 2 licenses of WatchDirectory or 2 licenses of WatchFTP during July 2009, you can get a free license of ShowSize, normally sold at $49.95.

To get your free copy of ShowSize, forward your WatchDirectory or WatchFTP order confirmation to support@watchdirectory.net. Please allow a few days to process your request.

About ShowSize

You already know that Windows Explorer does not show folder sizes in its list. ShowSize can do that and much more. See how your disk space is distributed in various folders and file types, and so on. ShowSize is the next generation software that can process terabytes of disk space and millions of files. Using ShowSize, you can also list or print folder contents in ways not possible with Windows Explorer.

YourSurprise.com - One of our Customers

YourSurprise.com is a company which sells personalized Gifts via the Internet. The personalized gifts mostly exist of a product, personalized with an image. For every order we create an order form in PDF format.

For our new website, released somewhere in 2008, we were looking for a program to automatically download the images and PDF’s and print these PDF’s on demand. From my previous internship I already knew WatchDirectory, PDF print on demand. Browsing the website of WatchDirectory I’ve noticed that they also had the program WatchFTP. So one company which solved our problems!

These images, created by the users, are automatically downloaded with WatchFTP to our local server where the gifts are created, handled and shipped. The order form PDF’s are also downloaded and printed on demand with WatchDirectory. We had some questions about tuning/configuring the products and within no time we had a reply with the solution.

In short, two top products which fit to our needs exactly.

   Niels van der Linde
   YourSurprise.com

Vlaggetjesdag

YES! Today in Holland we celebrate "Vlaggetjesdag" (day of the flags?). It is traditionally the day the first salted herrings of the new season are sold. Most Dutch eat the Herring with raw onions, these onions were originally added to camouflage the smell of rotting fish. These days they are not really needed anymore, and Holland is divided in 2 camps about adding onions (I like my herring with onions, thank you).

The herring is sold as "Hollandse Nieuwe" (Dutch New) when it is at least 16% fat (yummie!) and prepared in the traditional way (salted and "gekaakt"). The first barrel with herring is always sold at ridiculous prices (€66,000 this year), the money goes to charity.

I lived in Denmark for a year and sadly enough, they don't sell Herring prepared the "Dutch way". Luckily, their "Sild i Karry" (Herring with a lot of herbs and Curry sauce) was super as well - actually, besides Carlsberg Porter, it is one of the main reasons for my frequent visits to Denmark.

Remotely start/stop WatchDirectory and WatchFTP services

A great free tool from Sysinternals (now owned by Microsoft) is the psexec program. With psexec you can run programs on other computers and it supports full interaction with remote command prompts. This allows you, for example, to start or stop Windows Services by running the "NET START" or "NET STOP" command.

Initial Setup

I always create a new user on the computer I want to access with psexec and name it psexec as well. I give this user administrator privileges so it can do "whatever I want". This step may not be necessary, see the psexec help (just run psexec without parameters).

Control a Service on another machine

Okay, lets assume I want to stop the WatchDirectory task "unzipbiggies" that runs as a Service on computer \\Test01. First, you need to know that when WatchDirectory creates a Windows Service, the Service name is prefixed with "watchDirectory:", so the actual Windows Service name for the task "unzipbiggies" is "watchDirectory:unzipbiggies".
WatchFTP prefixes Service names with the string "WatchFTP:".

First we need to open a command prompt on the remote computer:

C:\Users\gert>e:\bin\psexec.exe \\Test01 -u psexec -p topsecret cmd

This assumes you created a user "psexec" with password "topsecret" on the \\Test01 computer.

After a while this opens a command prompt on the remote computer, but it shows on your computer. Enter the following command to see all running Windows Services (including the WatchDirectory and WatchFTP services):

C:\Users\gert>NET START
These Windows services are started:

... snipped a lot of services....
   Terminal Services
   Themes
   UPnP Device Host
   User Profile Service
   watchDirectory:unzipbiggies
   wdPostMan
   WebClient
   Windows Audio
   Windows Audio Endpoint Builder
... snipped a lot of services....
The command completed successfully.

To stop the WatchDirectory task, enter

C:\Users\gert>NET STOP "watchDirectory:unzipbiggies"

To start the task again, enter

C:\Users\gert>NET START "watchDirectory:unzipbiggies"

Other uses

There are a lot of other things you can do with psexec. For example when a file is detected by WatchDirectory on "this" computer, let psexec run a command on another computer. You get the idea, I guess.

WatchFTP 2.1.9

Today we released a minor update for WatchFTP. This free update has the following fixes/new features:

• A few more problems with accented characters were fixed
• In certain cases files were not removed from the FTP server
• The "History Viewer" allows you to show the full timestamps of messages

You can download the new version here:
http://www.watchftp.com/download.html
or http://www.watchftp.de/download.html
or http://www.watchftp.es/download.html

Stop all your tasks and install the new version "on top" of your current install so it will see your settings/tasks.

Let us not forget...

Our Brave Soldiers

Our heroes, Captain Kroon and his troops, they serve our country while we are sleeping. Soon (May 29, 2009) Captain Kroon will be awarded for his own and his troops' bravery.

Marco Kroon, the first person in a long time (50 years) in The Netherlands to receive the most honorable honor, our most prestigious medal: the Williams Order.
Marco, and your troops, you understand this is for all of you, so: THANK YOU!

This rare honor is only awarded for the highest bravery. Sometimes (extremely rare!) it is honored to a whole regiment. Even rarer to a foreign regiment, so far this has happened only twice in hundreds of years....

Our Polish Friends: 1st Independent Parachute Brigade
Our American Friends: 82nd Airborne Division

---

Update - I have just seen the ceremony on TV, very impressive!

(Marco Kroon being decorated by Queen Beatrix)
If you understand Dutch, here is a short interview with our hero, this was just after he heard he will be knighted.

---

Update - A short clip of the ceremony (~2 minutes). With a simple slap on the shoulder, Marco is knighted - did I tell you I love traditions?

How Many Tasks does WatchDirectory support?

A question that sometimes comes up is
How Many Tasks does WatchDirectory Support?

If this is for making a purchase decision (Do I need WD Standard or Professional), the answer is:
WD Standard supports 3 tasks. WD Professional supports..... mmmmm, well it depends.

Number of Tasks

There is no artificial limit for the number of tasks that WatchDirectory Professional can run. There are, however, other limitations (mostly memory related) that limit the number of tasks.

• Available Memory
  Each task you run requires some memory. The amount of memory needed depends on several things, most importantly on the Monitoring Method you use.
• Size of Monitored Directories
  If you use the "Default" or "Polling" Monitoring Method, your task will have a representation of the monitored directory in memory. Large directories require more memory. The "File Age" and "Fast" monitoring methods do not require this representation.
  See also Monitor Huge Directories.
• Run Mode
  If you run your tasks as a Windows Service, there are some Windows limitations for the number of Services supported. The actual limit depends on your Windows version and on the number of other services you have running. If you reach this limit, you will typically receive error messages like this:
  StartService fails: Error Code:1053
  Description: The service did not respond to the start or control request in a timely fashion.
  The last few posts in this forum topic describes a few work-arounds.
• The Size of your Brain
  Can you really manage 200 (WatchDirectory) tasks?

Why so Many?

Often it can be easy to replace several tasks by just one, especially if their monitored directory "share" a common parent directory. A few tips....

Replace multiple "Copy Files" by one "Sort Files" task

If you want to copy files to multiple destinations, you don't have to create multiple "copy files" tasks. Instead create just one "Sort Files" task. Each "rule" you create can have another destination.
See also The Sort Files plugin.

Merge multiple "Email Notification" tasks into one

When the email address is the only thing that's different for each task, you may want to look at the "Advanced Addressing" options for the email plugin. This allows you to specify different email addresses depending on the sub-directory where the file is found. An old newsletter also describes this.

Update (WatchDirectory 4.6.2): When you use Advanced Addressing, using the "email.txt" trick, you can also set the Subject and message body depending on the directory where the file is found.

See also The Email Plugin and Advanced Addressing.

Convert tasks into Subtasks

WatchDirectory v4.5.8 introduced a new plugin that can run other tasks as Subtasks. There are a few limitations described further in the help file.
See also The Run Subtasks Plugin.