A great way to process files in a Job is to use the 'Copy Files' step.
You can use this step in any number of ways, from post transform processing of data in ftp'd files to backup and archive directories, to building of simple utilities.
One of the gotchas that I see often is what to put in the wild card field. This field is a regular expression so if you are on a windows machine, and want all of the excel files in a directory, *.xls won't quite get you there.
Converting that to a regular expression would read: ^.*xls.
There are many sites out there that can help you spin up on regular expressions, and once you get the hang of them, can make your life quite a bit easier when dealing with strings and string processing.
In the context of string searching, happy hunting.
Doug W.
The article is so appealing. You should read this article before choosing the Google cloud big data services you want to learn.
ReplyDelete