Walkthrough: Telecommunication Company’s Use of AutoMate BPA Server Tackles their Big Data Problem

by Dallas Li, in Ask an ASE, posted 6/27/12
image for article

We discussed how a multibillion dollar cable telecommunications company uses AutoMate BPA Server to automate its mission-critical, daily process in last month’s blog post.

Today, we will look at a detailed outline of the workflow and task utilized to create this automation solution.

Let’s take a look at the company’s requirements again:

The company’s IT team is responsible for processing and finding error codes for more than 500,000 verification files each day. Error codes are needed to be found and examined from these files and the IT team also needs to log each error count and calculate the percentage of each error occurrence. After processing these error statistics, the IT team is required to upload and update the statistical report, then compress all the verification files by regions and archive to a network location for later viewing by corporate executives.

AutoMate Workflow

The workflow below covers the company’s requirements. Here, we use a schedule trigger to launch our workflow, which goes to a network folder to count and process the verification files. Once completed, it then calculates the percentages of errors and successes, and logs the statistics to a file. Finally, it emails the administrators for successful completion; otherwise, it notifies the administrators for errors.

Workflow

AutoMate Task

Since we have covered how to set up the schedule trigger and how to use the email action to send notifications in our previous Ask An ASE blogs, our walkthrough today will focus on the main task of the workflow: Process Verification Files and Create Stats File.

First, let’s take a look at the task.

Task

Variables

We have 10 variables in our task to populate the data for us. To create a variable, we can use the Create Variable action from the Variables Actions Folder in the Task Builder. Specify the name of the variable and click the OK button. We then repeat the same action for the rest of the variables.

Variable

Next, we would like AutoMate to process each verficiation file in a folder. For this, we use the Loop Files action from the Loop Actions Folder to go through each file in the folder.  Specify the path of the folder where the verification files are located.  Note that we are using the variables in this example:  %VAR_Directory% is a variable that stores the path of the directory folder, while %VAR_Folder% is a variable that stores the name of the folder of the verification files.  Since the Loop Files action looks at a file one at a time, we also need to specify a variable to populate the file name each time it loops. Select VAR_Filename to populate the file name and click the OK button.

Loop

Evaluation: Success or Error

Once we know the file name of a file, we want to examine each line of the file to look at errors and successes. Since it involves repeatedly reading a line over and over again, we also want to use loop in this step to repeat the action. For this, we are using the Loop File Content action from the Loop Actions Folder. In order for AutoMate to loop through each line of the file content, we need to specify the file path first. In this example, we are using the variables %VAR_Directory%\%VAR_Folder% to specify the folder of the file, while using %VAR_Filename%, which was populated in the previous Loop Files step to specify the file name.  Next, we specify %VAR_Line% to store the line content temporary each time it loops. Finally, we specify the delimiter. For this example, the file content is separated by a new line.

Loop Contents

Next, we parse the last four characters of each line (%VAR_Line%) in the file to get the status field.  In this example, we are using a VB function, Right() from the Expression Builder to get the last four characters. Note that there are alternative ways of doing this, such as the Find Text and Get Subtext actions.

Set Variables

After populating the VAR_StatusField, we come to an evaluation point to determine if the status is a success (0001) or a failure (anything other than 0001) by using the If action from the Flow Actions Folder. 

If

For the next few steps, we specify which actions to take based on the evaluation results. We want to increment the count of %VAR_SuccessCount% by one if %VAR_StatusField% is “0001”; otherwise increment the count of %VAR_ErrorCount% instead. Then we finish the condition by ending it with the End If step.

Condition

After getting the counts on %VAR_ErrorCount% and %VAR_SuccessCount%, we increment %VAR_TotalCount% by one to keep track of how many lines or entries we have processed. To do this, we use the Increment Variable action from the Variable Action Folder.

Increment Variable

Calculating Success or Error

Once we have all the counts, we need to calculate the percentages for the errors and successes. To do this, we also use the Set Variable action in conjunction with the Round() function from the Expression Builder. 

The long expression %(Round((VAR_ErrorCount / VAR_TotalCount)*100, 2)) & "%%"% may look a bit overwhelming, let’s dissect it together: 

  •  The first and last % signs: These indicate to AutoMate that it’s an expression to evaluate (just like how all the variables are also wrapped around with the % when we reference them). So now we are left with (Round((VAR_ErrorCount / VAR_TotalCount)*100, 2)) & "%%".  
  •  (VAR_ErrorCount / VAR_TotalCount): This is a division of two numbers — we are dividing the count of errors by the total count. With the quotient, we use *100 to multiply the quotient by 100 for the percentage. 
  •  Round function: This is used to round the percentage to the last two decimal places (2, in this case). To simplify this, let’s say the VAR_ErrorCount = 30 and VAR_TotalCount = 100, giving us this: %(Round((50 / 100)*100, 2)) & "%%"%

Our quotient in this example would be 0.5, so our expression would look like this: %(Round((0.5)*100, 2)) & "%%"%. 

After multiplying the quotient by 100, it becomes this: %(Round(50, 2)) & "%%"%. 

Rounding the final product to two decimal places would result in our expression looking like this: %50.00 & "%%"%. 

Then finally, the “%%” is for a literal % sign.  So the output of the expression would be 50.00%.

Percent

Once all the counts and percentages are gathered and calculated, we log the data to a file by using the Write to File activity from the File System action. First, specify the location and the file name to create the log file. Then specify the data to write.

Write File

Before it finishes the loop, we need to reset the variables %VAR_ErrorCount%, %VAR_SuccessCount% and %VAR_TotalCount% to 0 for the next verification file.

Reset Counts

That’s all it takes to automate this process. The AutoMate workflow discussed in this blog is responsible for processing and reading more than 500,000 verification files on a daily basis. The work that used to take a full working day with a team of 10 IT professionals is now fully automated, and the entire process is completed in the background in just about two hours, without any human intervention.