TwickrTape

Real-time scrolling Tweets related to financial news alerts and updates. Just enter the Twitter handle you want to see a marquee of live streaming tweets from.

TL;DR - the working app is here: https://twickrtape.azurewebsites.net
..znd here on Google Play Store: https://play.google.com/store/apps/details?id=io.cordova.twicktape

Use button in lower-left to toggle different Twitter handles

The code is a hodgepodge of various references and vestiges of past personal projects. My initial aim was to get something working end-to-end as I envisioned, and I was able to achieve that pretty easily through the Twitter API and .NET.

The process is standard API stuff- first you authenticate and receive a token which you can then subsequently pass to prove that your requests are valid. Then, using simple (REST API) HTTP GET, we get the content (Twitter Timeline information) that we are interested in.

The gist of the code is .NET C# (an ASP.NET Controller method) and listed below:

  public ActionResult GetTwickr(string handle = "business")  
     {  
       // Set your own keys and screen name  
       var oAuthConsumerKey = "XXXXXXXXXXXXX"; // "API key";  
       var oAuthConsumerSecret = "XXXXXXXXXXXXXXXXXXXXX"; // "API secret key";  
       var oAuthUrl = "https://api.twitter.com/oauth2/token";  
       var screenname = "@" + handle; // default Twitter display current status  
       // Authenticate  
       var authHeaderFormat = "Basic {0}";  
       var authHeader = string.Format(authHeaderFormat,  
         Convert.ToBase64String(Encoding.UTF8.GetBytes(Uri.EscapeDataString(oAuthConsumerKey) + ":" +  
         Uri.EscapeDataString((oAuthConsumerSecret)))  
       ));  
       var postBody = "grant_type=client_credentials";  
       HttpWebRequest authRequest = (HttpWebRequest)WebRequest.Create(oAuthUrl);  
       authRequest.Headers.Add("Authorization", authHeader);  
       authRequest.Method = "POST";  
       authRequest.ContentType = "application/x-www-form-urlencoded;charset=UTF-8";  
       authRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;  
       using (Stream stream = authRequest.GetRequestStream())  
       {  
         byte[] content = ASCIIEncoding.ASCII.GetBytes(postBody);  
         stream.Write(content, 0, content.Length);  
       }  
       authRequest.Headers.Add("Accept-Encoding", "gzip");  
       WebResponse authResponse = authRequest.GetResponse();  
       // deserialize into an object  
       TwitAuthenticateResponse twitAuthResponse;  
       using (authResponse)  
       {  
         using (var reader = new StreamReader(authResponse.GetResponseStream()))  
         {  
           System.Web.Script.Serialization.JavaScriptSerializer js = new System.Web.Script.Serialization.JavaScriptSerializer();  
           var objectText = reader.ReadToEnd();  
           twitAuthResponse = JsonConvert.DeserializeObject<TwitAuthenticateResponse>(objectText);  
         }  
       }  
       try  
       {  
         // Get timeline info  
         var timelineFormat = "https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name={0}&include_rts=1&exclude_replies=1&count=5";  
         var timelineUrl = string.Format(timelineFormat, screenname);  
         HttpWebRequest timeLineRequest = (HttpWebRequest)WebRequest.Create(timelineUrl);  
         var timelineHeaderFormat = "{0} {1}";  
         timeLineRequest.Headers.Add("Authorization", string.Format(timelineHeaderFormat, twitAuthResponse.token_type, twitAuthResponse.access_token));  
         timeLineRequest.Method = "GET";  
         WebResponse timeLineResponse = timeLineRequest.GetResponse();  
         var timeLineJson = string.Empty;  
         string scrolltxt = string.Empty;  
         using (timeLineResponse)  
         {  
           using (var reader = new StreamReader(timeLineResponse.GetResponseStream()))  
           {  
             timeLineJson = reader.ReadToEnd();  
           }  
         }  
         // deserialize into an object  
         dynamic obj = JsonConvert.DeserializeObject(timeLineJson);  
         foreach (var r in obj.Root)  
         {  
           scrolltxt += " ***** " + r.text;  
         }  
         var model = new LoggedInUserTimelineViewModel { TimelineContent = scrolltxt, Handle = handle };  
         return View("TwickrMain", model);  
       }  
       catch (Exception e) { }  
       return View("TwickrMain");  
     }  

Next, using Newtonsoft,Json JsonConvert() function, we deserialize the JSON response and zero in on the 'text' property of each entity (tweet) in the array of JSON results (tweets).

And finally, using some basic Bootstrap HTML, JavaScript and CSS we are able to wire up a very simple, but effective app that interacts with the Twitter API in real-time.

Reference: https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html

Using PowerShell and youtube-dl to automate grunt work

Earlier this evening I was tasked by my wife to create audio files from some (DRM-free) YouTube content. I knew how to do this via a manual process, however after the 5th manual file conversion I started to remember the youtube-dl Python project and figured there has to be an easier and quicker way to do this file conversion grunt work.

And sure enough... youtube-dl (with some help from PowerShell) does the trick.

Expected input and output

For starters (we are assuming you have already installed Python 3), you will need to have pip install two modules:

 pip install youtube-dl  
 pip install ffmpeg  

Next, create a youtube-uris.txt with the URI of each song you want to convert on separate lines of the file.

Finally, you can use the following PowerShell script to copy the song URIs into a string array which is then iterated through, converting each array item into a playable audio file (mp4), saved in whatever directory you run the script from.

 [string[]]$arrayFromFile = Get-Content -Path 'C:\sand\youtube-dl-sandbox\youtube-uris.txt'  
 foreach ($uri in $arrayFromFile) {  
   youtube-dl -f 140 $uri  
  }  

That's it. This script can be used for automating other manual/repetitive tasks requiring a procedure to be run against every item in a large list.


Reference: https://github.com/rg3/youtube-dl

Generate Secure Machine Key Section for Web.config via PowerShell

Machine Keys are used in ASP.NET for securing machines that are part of a web farm as well as for sharing encrypted application session and state information.

This PowerShell script (function) can be called (once run and saved to your PS session) via,

"PS C:\: Generate-MachineKey"


With the output from this PS function, you can copy and paste to your web.config ie:
 <configuration>  
  <system.web>  
   <machineKey ... />  
  </system.web>  
 </configuration>  

Generate-MachineKey function definition/PS script
 # Generates a <machineKey> element that can be copied + pasted into a Web.config file.  
 function Generate-MachineKey {  
  [CmdletBinding()]  
  param (  
   [ValidateSet("AES", "DES", "3DES")]  
   [string]$decryptionAlgorithm = 'AES',  
   [ValidateSet("MD5", "SHA1", "HMACSHA256", "HMACSHA384", "HMACSHA512")]  
   [string]$validationAlgorithm = 'HMACSHA256'  
  )  
  process {  
   function BinaryToHex {  
     [CmdLetBinding()]  
     param($bytes)  
     process {  
       $builder = new-object System.Text.StringBuilder  
       foreach ($b in $bytes) {  
        $builder = $builder.AppendFormat([System.Globalization.CultureInfo]::InvariantCulture, "{0:X2}", $b)  
       }  
       $builder  
     }  
   }  
   switch ($decryptionAlgorithm) {  
    "AES" { $decryptionObject = new-object System.Security.Cryptography.AesCryptoServiceProvider }  
    "DES" { $decryptionObject = new-object System.Security.Cryptography.DESCryptoServiceProvider }  
    "3DES" { $decryptionObject = new-object System.Security.Cryptography.TripleDESCryptoServiceProvider }  
   }  
   $decryptionObject.GenerateKey()  
   $decryptionKey = BinaryToHex($decryptionObject.Key)  
   $decryptionObject.Dispose()  
   switch ($validationAlgorithm) {  
    "MD5" { $validationObject = new-object System.Security.Cryptography.HMACMD5 }  
    "SHA1" { $validationObject = new-object System.Security.Cryptography.HMACSHA1 }  
    "HMACSHA256" { $validationObject = new-object System.Security.Cryptography.HMACSHA256 }  
    "HMACSHA385" { $validationObject = new-object System.Security.Cryptography.HMACSHA384 }  
    "HMACSHA512" { $validationObject = new-object System.Security.Cryptography.HMACSHA512 }  
   }  
   $validationKey = BinaryToHex($validationObject.Key)  
   $validationObject.Dispose()  
   [string]::Format([System.Globalization.CultureInfo]::InvariantCulture,  
    "<machineKey decryption=`"{0}`" decryptionKey=`"{1}`" validation=`"{2}`" validationKey=`"{3}`" />",  
    $decryptionAlgorithm.ToUpperInvariant(), $decryptionKey,  
    $validationAlgorithm.ToUpperInvariant(), $validationKey)  
  }  
 }  

Accessing SQL Server Data in R

Importing SQL Server data into R for analysis is pretty straightforward and simple. You will obviously need R installed on your machine. The following R code will connect to your SQL Server database (using R Studio):

 library(RODBC)  
 dbconnection <- odbcDriverConnect('driver={SQL Server};server=.;database=CLARO;trusted_connection=true')  
 initdata <- sqlQuery(dbconnection,paste('SELECT * FROM [CLARO].[dbo].[Fielding];')) 


SELECT data from a SQL Server database output in R Studio


Accessing SQL Server Data in Python

So you want to access Microsoft SQL Server from your Python script(s)?

After weeding out some long-abandoned and/or nonworking solutions, I discovered a very simple Python ODBC driver that works with virtually all SQL Servers since MSSQL 2005 called "pyodbc". 

First, you will need to install this MSSQL ODBC (13.1 or 17 should work) component on your machine in addition to installing the pyodbc driver.

Next, get the pyodbc module for Python by running this from Windows command prompt:

pip install pyodbc

Then open up a python shell using 'py' or 'python' and enter the following after editing configuration values to match your development environment:

 import pyodbc  
 cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER=localhost;DATABASE=WideWorldImporters;UID=DemoUser;PWD=123Password')  
 cursor = cnxn.cursor()  
 #Sample of a simple SELECT  
 cursor.execute("SELECT TOP (100) Comments, count(*) FROM WideWorldImporters.Sales.Orders GROUP BY Comments")  
 row = cursor.fetchone()   
 while row:   
   print(row[0] + ': ' + str(row[1]))  
   row = cursor.fetchone()  

Running this code will result in the below if you have configured everything correctly (note this example makes use of the Microsoft SQL Server demo WorldWideImporters database):


Reference: https://docs.microsoft.com/en-us/sql/connect/python/pyodbc/step-3-proof-of-concept-connecting-to-sql-using-pyodbc?view=sql-server-2017

.udl for DbConnection Check

This is a useful method to quickly check SQL credentials and/or RDBMS connectivity if working on a Windows OS. Just create a file in any editor (ie. Notepad) and save it with .udl extension which makes it a Microsoft Data Link file type. Then, right-click and inspect file Properties >> "Connection" tab.

Credit to a former colleague of mine (thanks Gene David!) who showed me how to use this simple but very useful trick.



Short Selling

Broker borrows a share, sells the share high, repurchases share at lower price ($) and returns it.


Short selling stock is the practice by which a broker borrows stock with the hope that the price of that stock will fall so that he or she can sell at a high price, (re)purchase at a lower price, and pocket the difference.

Hypothetically, let's say a trader named Joe firmly believed that Apple, Inc. was about to experience a large drop in share price. To short a single share of Apple stock, Joe would do the following:

1). Borrow a share of APPL from his portfolio, a client portfolio, or a fellow broker
2). Sell the share at the highest a price they can find before a drop (say 1 share of AAPL at current $157.76)
3). Wait for the price to fall (say APPL falls to $102.76), then purchase one share at this lower price
4). Subtract the higher price from the lower price (less fees) and return the borrowed share. 
Joe earns a cool $53 bucks from this scheme as he sold at $157.76 and bought back for just $102.76. After fees of $2.00 this is $157.76 - $102.76 -$2.00 == $53.00.

While the idea of selling something short of true value is often associated with the nefarious case of a stock "short" like this, oftentimes it is a necessity. The market always needs people on both the long end (owners/buyers) and the short end (renters/sellers) for it to work properly.

This is why banks who are on the hook with a property that they cannot sell will ultimately agree to a "short sale" (selling the home for below its fair market value) to recoup at least some of their losses.

A combination of consumer preferences and financial factors determine whether to go long or short on any kind of investment or large financial transaction.



Short selling doesn't always work in the sellers favor

Refactoring Made Obvious

Refactoring (as a term if not as a practice) gets thrown around quite a bit. Sometimes really necessary refactoring doesn't get the priority it deserves because it is hard to quantify or even visualize/sense the result of a good refactoring (it should be relatively transparent in experience to the original, any differences should be optimization or enhancement- without losing of the original functionality along the way). We can take the case of a simple JavaScript animation, for example.

Simple animating of "+" dropping across the browser screen

Long ago, I used a mobile app that had a neat UI animation feature I really liked, but it took me a while to track down just how to accomplish it. I found some good starting points on SO, and began implementing a draft on JSFiddle.net.

The animation behavior I went about creating is simply a delay of an HTML element falling from the screen (via the "topToBottom" variable you'll see below- which is just the browser screen height property) . In a cascading and sequential set of delays, each falling element is pushed an increasing distance from the left of the screen so that the elements can fall independently (otherwise you would see just one column for all of the falling +'s in the end result).

In the following 4 steps, I am going to present a very basic refactoring scenario- going from the code template references I found, to the final refactored code (css, html and .js).

(1) First I found a fiddle via the SO question reference below: http://jsfiddle.net/reWwx/4/

(2) I then changed the code to create my own draft on JSFiddle.net: http://jsfiddle.net/reWwx/539/

(3) Next, I created an .htm file with the CSS styles and JavaScript inline (not quite what we'd want to check into source control...):
 <html>  
 <head>  
 <style>  
 body {height: 600px; background-color: #999}  
 #line-3 {  
   position:absolute;  
   width:100%;  
   left:20px;  
   top:0px;  
 }  
 #line-4 {  
   position:absolute;  
   width:100%;  
   left:30px;  
   top:0px;  
 }  
 #line-5 {  
   position:absolute;  
   width:100%;  
   left:40px;  
   top:0px;  
 }  
 #line-6 {  
   position:absolute;  
   width:100%;  
   left:55px;  
   top:0px;  
 }  
 #line-7 {  
   position:absolute;  
   width:100%;  
   left:70px;  
   top:0px;  
 }  
 #line-8 {  
   position:absolute;  
   width:100%;  
   left:85px;  
   top:0px;  
 }  
 #line-9 {  
   position:absolute;  
   width:100%;  
   left:100px;  
   top:0px;  
 }  
 #line-10 {  
   position:absolute;  
   width:100%;  
   left:115px;  
   top:0px;  
 }  
 #line-11 {  
   position:absolute;  
   width:100%;  
   left:130px;  
   top:0px;  
 }  
 #line-12 {  
   position:absolute;  
   width:100%;  
   left:145px;  
   top:0px;  
 }  
 #line-13 {  
   position:absolute;  
   width:100%;  
   left:160px;  
   top:0px;  
 }  
 #line-14 {  
   position:absolute;  
   width:100%;  
   left:175px;  
   top:0px;  
 }  
 #line-15 {  
   position:absolute;  
   width:100%;  
   left:195px;  
   top:0px;  
 }  
 #line-16 {  
   position:absolute;  
   width:100%;  
   left:210px;  
   top:0px;  
 }  
 </style>  
 <script>  
 $(document).ready(function(){  
   var bodyHeight = $('body').height();  
   var footerOffsetTop = $('#line-3').offset().top;  
   var topToBottom = bodyHeight -footerOffsetTop;  
  $('#line-3').css({top:'auto',bottom:topToBottom});  
  $("#line-3").delay(100).animate({  
   bottom: '100px',  
   }, 2200);   
  $('#line-4').css({top:'auto',bottom:topToBottom});  
  $("#line-4").delay(108).animate({  
   bottom: '100px',  
   }, 2200);   
  $('#line-5').css({top:'auto',bottom:topToBottom});  
  $("#line-5").delay(145).animate({  
   bottom: '100px',  
   }, 2200);   
  $('#line-6').css({top:'auto',bottom:topToBottom});  
  $("#line-6").delay(119).animate({  
   bottom: '100px',  
   }, 2200);   
  $('#line-7').css({top:'auto',bottom:topToBottom});  
  $("#line-7").delay(115).animate({  
   bottom: '100px',  
   }, 2200);   
    $('#line-8').css({top:'auto',bottom:topToBottom});  
  $("#line-8").delay(176).animate({  
   bottom: '100px',  
   }, 2100);   
    $('#line-9').css({top:'auto',bottom:topToBottom});  
  $("#line-9").delay(13).animate({  
   bottom: '100px',  
   }, 2200);   
    $('#line-10').css({top:'auto',bottom:topToBottom});  
  $("#line-10").delay(12).animate({  
   bottom: '100px',  
   }, 2200);   
    $('#line-11').css({top:'auto',bottom:topToBottom});  
  $("#line-11").delay(11).animate({  
   bottom: '100px',  
   }, 2000);   
    $('#line-12').css({top:'auto',bottom:topToBottom});  
  $("#line-12").delay(10).animate({  
   bottom: '100px',  
   }, 2100);   
    $('#line-13').css({top:'auto',bottom:topToBottom});  
  $("#line-13").delay(11).animate({  
   bottom: '100px',  
   }, 600);   
    $('#line-14').css({top:'auto',bottom:topToBottom});  
  $("#line-14").delay(14).animate({  
   bottom: '100px',  
   }, 700);   
      $('#line-15').css({top:'auto',bottom:topToBottom});  
  $("#line-15").delay(14).animate({  
   bottom: '100px',  
   }, 800);   
      $('#line-16').css({top:'auto',bottom:topToBottom});  
  $("#line-16").delay(24).animate({  
   bottom: '100px',  
   }, 900);   
 })  
 </script>  
 </head>  
 <body>  
 <div id="line-3">+</div>  
 <div id="line-4">+</div>  
 <div id="line-5">+</div>  
 <div id="line-6">+</div>  
 <div id="line-7">+</div>  
 <div id="line-8">+</div>  
 <div id="line-9">+</div>  
 <div id="line-10">+</div>  
 <div id="line-11">+</div>  
 <div id="line-12">+</div>  
 <div id="line-13">+</div>  
 <div id="line-14">+</div>  
 <div id="line-15">+</div>  
 <div id="line-16">+</div>  
 </body>    
 </html>  
(4) And lastly I identified all of the repeating parts and made them dynamic in JavaScript, using the jQuery library to shorten much of the .js behavior:
  <html>   
  <head>   
  <style>   
  body {height: 600px; background-color: #000000; color:lime;}   
  div {   
   position:absolute;   
   top:0px;   
    width:100%;   
  }   
  </style>   
  <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>   
  <script>   
  $(document).ready(function(){   
   var base = $('#base');  
   var topToBottom = $('body').height();    
   for(i=0; i<83; i++){   
    base.append('<div id=\"line-'+i+'\" style=\"left:'+ Math.abs(i*10) +'\px">+</div>')   
    $("#line-" + i).css({top:'auto', bottom:topToBottom}).delay(100*i).animate({bottom: '100px'}, (1000));    
   }   
  })   
  </script>   
  </head>   
  <body id="base">   
  </body>    
  </html>   

The final result contains the same behavior of the draft but it eliminates repetition by dynamically generating the HTML and dynamically attaching the falling action (which is really just coordinated position and visibility property changes behind the scenes of .animate()). Eliminating duplication, standardizing for better readability, reorganization for better clarity of code purpose and finding patterns (or finding different patterns that are a better match for the task) are the key concepts in refactoring.

Try it yourself by copying the code above, saving to an .htm file and opening the file in a web browser.

Final JSFiddle result: https://jsfiddle.net/radagast/6uzypc80/5

Larger font is always fun: https://twickrtape.azurewebsites.net/Home/About

Reference: https://stackoverflow.com/questions/8518400/jquery-animate-from-css-top-to-bottom

JavaScript for Progress on Scrolling

This topic is a prime example of why I write this web log. I've seen this functionality on countless web pages and mobile apps, but for some reason it is not well explained in most of the areas you will likely first wind up when searching for instructions on how to do this (if you wound up here first great, yay me).

Potential applications of progress on scroll might be a "Terms and Conditions" view, a code walk-through, etc.

The key code is all in $(window).scroll(function(). You have these 3 main components:

  • Document Height: $(document).height() === height of "viewable" window aka viewport
  • Window Height: $(window).height() === height of document being rendered
  • Scroll Top: $(window).scrollTop() === number of pixels content is scrolled

With these values you can set the width of the progress bar (the div with the class "scroll-progress-container"). The width is a simple calculation: scrollTop / (docHeight - windowHeight) * 100.

So you can attach this logic to an anonymous function within the browser's window scroll event a la: $(window).scroll(function() { ...

And then simply assign the result of  "$(window).scrollTop() / (docHeight - windowHeight) * 100;" to the width property of your <div class="scroll-progress">.

That's it.

Code:

 <!doctype html>  
 <head>  
 <title></title>  
 <style>  
 .header-container {  
  width: 100%;  
  height: 60px;  
  background-color: white;  
  position: fixed;  
  z-index: 10;  
  top: 0;  
  left: 0;  
 }  
 .header {  
  padding-left: 10px;  
 }  
 h1 {  
  margin-top: 15px;  
 }  
 .scroll-progress-container {  
  width: 100%;  
  height: 5px;  
  background-color: white;  
  top: 55px;  
  position: absolute;  
 }  
 .scroll-progress {  
  width: 0px;  
  height: 5px;  
  background-color: purple;  
 }  
 .filler-text {  
  width: 60%;  
  margin-top: 80px;  
  margin-left: 50px;  
  position: absolute;  
 }  
 </style>  
 <script src="https://code.jquery.com/jquery-1.11.2.min.js"></script>  
 <script>  
  $(document).ready(function() {  
    var docHeight = $(document).height(),  
    windowHeight = $(window).height(),  
    scrollPercent;  
     $(window).scroll(function() {  
       scrollPercent = $(window).scrollTop() / (docHeight - windowHeight) * 100;  
       $('.scroll-progress').width(scrollPercent + '%');  
     });  
 });  
 </script>  
 </head>  
 <body>  
 <div class="header-container">  
 <div class="header">  
 <h1>Progress Scroll Bar Example</h1>  
 </div>  
 <div class="scroll-progress-container">  
 <div class="scroll-progress"></div>  
 </div>  
 </div>  
 Enter lots and lots of text here...  
 </body>  
 </html>  


References

https://www.veracode.com/blog/managing-appsec/building-scroll-progress-bar-javascript-and-jquery

https://stackoverflow.com/questions/14035819/window-height-vs-document-height

DNS Rebinding and the Fallacy of "Walled Gardens"

Attacking Private Networks from the Internet with DNS Rebinding

The well-written research article above by Brannon Dorsey is a must-read for any developer of home-integrated devices, and perhaps, all developers, even those who develop products that run within highly secure networks that, via an authenticated user- (as was done with Stuxnet)- can inadvertently intercept and execute malicious code.

Don't let script kiddies mess with your code via DNS

Essentially Mr. Dorsey discovered that smart home gadgets that are supposed to operate securely in private networks can be exploited from the outside by simply embedding scripts containing network hi-jack exploits in links that can make requests back to the client browser that appear- via dynamic IP-to-same-origin-hostname switching* (ie. "DNS Spoofing") done through a malicious DNS server -to be from a trusted, same-origin source.

ie.

Malicious hostname: exploit.net
Malicious ip address: 59.33.12.9

Victim recent request hostname: somebank.com
Victim recent request ip address: 122.76.21.19

<<Very brief DNS Hijack via a malicious DNS server>>

Malicious hostname: exploit.net
Malicious ip address as far as victim browser client is aware: 122.76.21.19

......See the problem?

He goes on to explain how entire protocols like UPnP "are built around the idea that devices on the same network can trust each other".

Remember: what appears to the browser client to be a "same-origin" request is not always actually a same-origin request. Make sure that you change the default credentials on your network router(s) and as the article above insists:

"We need developers to write software that treats local private networks as if they were hostile public networks. The idea that the local network is a safe haven is a fallacy. If we continue to believe it people are going to get hurt."

Good to remember: protocol://host:port/path?query

*DNS cache poisoning, also known as DNS spoofing, is a type of attack that exploits vulnerabilities in the domain name system (DNS) to divert Internet traffic away from legitimate servers and towards fake ones.

Visualize Hashing and Salt as Part of Password Encryption Process

The image below is a simplified and easy-to-understand illustration of how hashing and salting work. The main takeaway from this post- multiple users can have the same password, but will all have different salt values, thus making their hash result value different, and when you authenticate, you authenticate by the hash result value of your passwords, which is virtually always going to be unique for each user record:

Simple, no?

Even in the case of 2 users having the same hash result, the usernames will/should not be the same, so you still have distinct accounts, because UserID is also checked in the authentication process.

Companies increasingly (and for good data privacy reasons) do not even store the clear text textbox value you enter when you sign up for and then log into Fb, Google, Amazon, etc- they check your entered password's hash result against the hash result they have for your user/account record either from when you registered or last changed your password.

Good answer to the question you may come across, "what is the difference between salt and an IV (initialization vector)?" (TL;DR: not all IV's are salt, but salt is a kind of IV): https://security.stackexchange.com/questions/6058/is-real-salt-the-same-as-initialization-vectors


Quality Control

You should know at least the surface topics surrounding TQM (Total Quality Management) because nearly all modern businesses practice TQM strategies and tactics to reduce costs and ensure top quality.

But first, check out this old video clip of America discovering something that ironically, an American (W. Edwards Deming) exported to Japan with great success years before:

1980 NBC News Report: "If Japan Can, Why Can't We?"

So big-Q "Quality" became a bit hit and has been embedded in process management throughout the globe ever since.

I think he has a point here.

Here are some Quality buzz words that surely you've heard before:

ASQ - American Society for Quality

"Black Belt" - Ooo. Ahh. It does mean something. It means a person has passed a series of very difficult exams on statistics and statistical process control for quality based on the quantitative technics and measures originated in Japan by W. Edwards Deming.

ISO 9001 - the International standard of a Quality Management System that is used to certify that business processes follow standard process and product guidelines.

Kaizen - a long-term approach to work that systematically seeks to achieve small, incremental changes in processes in order to improve efficiency and quality.

Kanban -  a visual system for managing work as it moves through a process.

Lean - a synonym for continuous improvement through balanced efficiency gains.

Example of statistical process control using UCL and LCL boundaries and a process (Fall Rate) improving.

LCL*  - Lower Control Limit - The negative value beyond which a process is statistically unstable.

MAIC - Measure, Analyze, Improve, Control.

Service Level Agreements (SLA) - A contract between a service provider and end user that defines the expected level of service to the end user.

UCL* - Upper Control Limit - The positive value beyond which a process is statistically unstable.

Uptime - Uptime is a measure of the time a service is working and available and opposite of Downtime.

Six Sigma - a statistical approach to process improvement and quality control; sometimes defined as +/-3 three deviations for the mean ("6"), sometimes as +/-6 deviations from mean.

The table above gives you an idea of realistic process improvement numbers (66,800 == a lot of defective items)


History and W. Edwards Deming
Quality Management is a permanent organizational approach to continuous process improvement. It was successfully applied by W. Edwards Deming in post-WWII Japan. Deming's work began in August 1950 at the Hakone Convention Center in Tokyo, when Deming delivered a speech on what he called "Statistical Product Quality Administration".

He is credited with helping hasten Japanese recovery after the war and then later helping American companies embrace TQM and realize significant efficiency and quality gains.


Deming's 14 Points for Total Quality Management

*Measures such as standard deviation and other distribution-based statistics determine the LCL and UCL for a process (any process- temperature of a factory floor, time to assemble a component, download/upload speed, defects per million, etc.).

References:

http://asq.org/learn-about-quality/total-quality-management/overview/deming-points.html

https://www.quora.com/How-did-W-Edwards-Deming-influence-Japanese-manufacturing

The GOF Software Design Patterns at a Glance

If you are serious about developing software, read this book. It contains timeless concepts you should know well.

I. Creational Patterns
1. Abstract Factory - facilitates the creation of similar objects that share interface
2. Builder - dynamic SQL statements and other dynamically constructed code
3. Factory Method - custom way to construct an object (vs. normal/default class constructor)
4. Prototype create new instances by copying existing instances
5. Singleton - assurance of 1 and only 1 instance of an entity in an application

II. Structural Patterns
1. Adapter - a means of making an object compatible with another object via a common interface
2. Bridge - the decoupling of the adaptor/interface and the incompatible items (light switch w/fans)
3. Composite a group of objects treated the same way as a single instance of the same type of object
4. Decorator - adding new functionality to a class via subclass with new methods, props, etc.
5. Façade - simplified interface to a complicated backend
6. Flyweight - reuse of an object instance to create more instances of the same type
7. Proxy - entity serving as a stand-in for the real thing. Often a wrapper for a more complex class.

III. Behavioral Patterns
1. Chain of Responsibility - source command is served by multiple processes with distinct linear roles.
2. Command - encapsulate a request as an object, works great with HttpRequest and user actions
3. Interpreter - dynamic culture and localization settings to display appropriate UI text
4. Iterator - centralize communication among objects that have a common interface (ie. TV remote)
5. Mediator - centralize complex communication and control among related objects
6. Memento - persist and recall object state
7. Observer - a central watcher picks up picks up broadcast messages from everywhere it observes (Twitter)
8. State - persists object state and can react in different ways according to attributes of its current state
9. Strategy - enabling an object to change/choose algorithm(s) at runtime
10. Template Method - great for interfaces; design skeleton w/base needs; allow concretes to implement details
11. Visitor - code execution varies according to visiting object (ie. keyboard keys, mice clicks, WiFi Routers, etc.)


References:

https://softwareengineering.stackexchange.com/questions/234942/differentiating-between-factory-method-and-abstract-factory

https://www.cs.cmu.edu/~charlie/courses/15-214/2016-spring/slides/24%20-%20All%20the%20GoF%20Patterns.pdf

https://quizlet.com/129434321/design-patterns-gof-in-1-sentence-flash-cards/

Encryption in Transit, Encryption at Rest

For example, the typical route of encryption is data being encrypted when a value is stored to disk or sent over a network, then that data is decrypted by an authorized shared key holder application or service, and once the decrypted data has been used, it is invalidated or (if modified) encrypted and stored back on disk or over and across networks to eventually be stored on disk, in user privacy-respecting, encrypted format.

Encryption in Transit
You want to encrypt input, let's say a credit card number, from someone's cell phone. You need the credit card number to be encrypted while en route but ultimately decrypted after the transit is complete and the encrypted card data arrives at the merchant server. Here is what encryption in transit looks like:

Data securely encrypted on the wire going to and fro...
Encryption at Rest
Take for example the case of storing that same credit card data on the merchant's server so that the credit card information can be reused for future purchase payments. In this case, you will want to keep the data encrypted when you write it to disk in order to preserve user privacy.

The data is only decrypted when it is necessary (ie. when a new payment is processed, the encrypted data will be briefly decrypted so that it can be sent to the payment processor service).

Data safely encrypted on disk storage

Encrypted data is only ever decrypted on demand- when something requests it. Encrypted data is secure so long as only intended parties have the shared secret(s) key(s) to decrypt the messages.


Reference: http://blog.fourthbit.com/2014/12/23/traffic-analysis-of-an-ssl-slash-tls-session

OLAP: Facts and Dimensions

OLAP can adequately be described as the storage of redundant copies of transactional records from OLTP and other database sources. This redundancy facilitates quick lookups for complex data analysis because data can be found via more and quicker (SQL execution) paths than normalized OLTP. And OLTP after all- should stick to its namesake and what it does best: processing, auditing, and backing up online transactions, no? Leave data analysis to separate OLAP Warehouses.

OLAP data is typically stored in large blocks of redundant data (the data is organized in ways to optimize and accelerate your data mining computations). Measures are derived from the records in the fact table and dimensions are derived from the dimension tables.


Facts are "measurements, metrics or facts of a particular process" -Wikipedia. Facts are the measurement of the record value: "$34.42 for today's closing stock price", "5,976 oranges sold at the market", "package delivered at 4.57pm", etc.

Dimensions are "lists of related names–known as Members–all of which belong to a similar category in the user’s perception of a data". For example, months and quarters; cities, regions, countries, product line, domain business process metrics, etc.

Dimensions give you a set of things that you can measure and visualize in order to get a better pulse and better overall understanding of the current, past and even potential future (via regression models) shape of your data- which can often alert you to things you might not be able to see (certainly not as quickly or easily) in an OLTP-based Data Analysis model which is often tied to production tables out of necessity or "we don't have time or money to implement a Data Warehouse".


Yes, you definitely do need OLAP/DW capabilities if you think long-term about it. Having intimate knowledge of your operations and how you can change variables to better run your business? Why would any business person (excepting those committing fraud) not want that?

I'd say that implementing a true and effective OLAP environment is worth any project investment and would pay itself over and again in the way of better and more specific/actionable metrics that help administrators of operations make the best, data-backed decisions- some very critical decisions involving millions of dollars and sometimes lives. I'd like a better look at the data before making a multi-million dollar or life/death decision.


SAS, Hadoop, SSAS, with healthy doses of R and/or Python customization?- whatever data solution you choose, my advice is to go with something that has a great track record of providing the kind of solutions that your business and/or your industry require. Use the tool that your ETL developers can utilize effectively and that helps you to best meet the needs of your constituents with whom you will exchange data with (the company, the customer, industry and/or regional regulation compliance).

Open Systems Interconnection (OSI) Model

"The purpose of the OSI reference model is to guide vendors and developers so the digital communication products and software programs they create will interoperate, and to facilitate clear comparisons among communications tools." -Vikram Kumar

Abstract OSI Model

OSI Model cross-references with the protocols that facilitate network communication

The OSI model is a visualized and defined "meaning" of abstract data communication across networks. Some people have argued that the OSI model doesn't meet the needs to explain the abstract ideas in modern TCP:
"To make matters worse, the Internet's evolution, based on TCP/IP, never strictly followed the old OSI model at all. A reasonable person might ask whether people who talk about 'Layer 1' or 'Layer 3' aren’t blowing kisses at an old friend instead of recognizing the relevance of the original OSI model."
Ouch. Well, like it or not the OSI model (10 years on after this 2008 article was published) is still one of the best ways we as IT people have to describe network packet travel from wire to UI screen and back.

Any analogy or model works so long as we understand what happens when we send web requests out to networks (by clicking, by pressing enter, by having certain background services sending random telemetry, polling and other non-user issued web requests).

But standardization is important for globally agreed-upon understandings that serve as a common template and enable "apples to apples" communication on ideas being researched and the building of solutions worldwide, which, at this juncture in the 21st Century, is increasingly common.

It truly is a small world after all and I'd argue we really do neeeed to have standards like:

ISO 35.100 - Open Systems Interconnection
ISO 9001 Quality Management
ISO/IEC 27001 Information Security Management,
ISO 14001 Environmental Management
ISO 31000 - Risk management

...in order to communicate in a common language across the earth and prevent a sprawl of  "20 ways to describe the same idea". Let's try to keep these things simple, standardized, and open to easy integrative communication.

References:

https://www.iso.org/standards.html

https://www.lifewire.com/layers-of-the-osi-model-illustrated-818017

https://www.networkworld.com/article/3239677/lan-wan/the-osi-model-explained-how-to-understand-and-remember-the-7-layer-network-model.html

Fire Drills to Ensure Successful Disaster Recovery

TEST. YOUR. DISASTER. RECOVERY. PLAN.

This cannot be said enough. I have heard countless horror stories (and lived 1) of instances where backups were completing "successfully" but no one ever tested the restoration of those backups (which were, in fact, corrupt and not restore-able)- until it was too late. :/


Put in the time up front to make sure your data recovery strategy and high-availability guarantee actually "recover" and make "available" your data- the lifeblood of your organization.

It is important to remember that you can have a Secondary database server that is 100% in sync with the Primary database server, but if your application is not configured to make the switch (or your service host does not make the switch for you)- you will only have 1/2 of things recovered: the data, but not the data availability through the application.

Good References on Disaster Recovery Strategies and what to consider ("sensitivity of data, data loss tolerance, required availability, etc."):

https://www.sqlshack.com/sql-server-disaster-recovery/

https://support.microsoft.com/en-us/help/822400/description-of-disaster-recovery-options-for-microsoft-sql-server


ETL and the Case Against a Single System for All Business Processes

Nearly every business wants to centralize their business data so that they can get an at-a-glance overview of what is going on at any given point in the day. Suppose, for instance, a company has its business data on 5 separate database servers, in various databases therein, across several different tables. And much of the data will need to be slightly modified (transformed) in order to make data types consistent and optimize it for quick and meaningful data analysis.

Assume the database servers are:
  • CMS, Orders (Dynamics 365 data)
  • Accounting (Sage 50 data)
  • Warehouse, Logistics (SAP data)
  • Field Service, Product Support (SalesForce data)
  • Call Center (TantaComm data)
Furthermore, the business wants to apply some custom logic to various fields so that certain calculations don't have to be made by the OLAP end users and applications.

ETL is just moving data from a Source to a Destination, often making some changes (transformations) along the way

Well, it all seems so complicated, right? How can all that disparate data be combined and standardized? It's not as complicated as allowing an ERP solution define how your various departments do their job. Some ERP products truly do have the functionality to satisfy 80-85% of the company business requirements. And when that is the case, by all means, companies should go that route.

But for instances where ERP of every business function is not working, you can use products like SSIS, Informatica, and Oracle Data Integrator to easily map ETL data visually and apply code operations to the data using a variety of programming languages.

But of course, there is no "one-size-fits-all 100% of our business needs" ERP solution; different business productivity software work better or worse for different industries, different departmental needs, for all manner of different reasons. Why care about the choice of business tool that your business' management and best minds know and have proven is most effective for their- often very specific- job?

Allowing a business to integrate the "best apps for each particular type of job" is nothing more than a sort of microservices architecture. But if we are working with custom code in these various LOB applications (at least some of the LOB apps will need custom API integrations so that they can talk to each other), with each new block of custom code, comes more decentralized code bases and decentralized test scripts...

All considered, I would argue that trying to put square pegs into round holes a la "ERP-everything" is a much, much, MUCH bigger effort than a well-developed and tested scheduled ETL Data Warehouse refresh process to centralize your company's data.

It's always a balance with software. Use the most effective apps for your organization and departmental requirements, but try to avoid the dependency hell that can come along with a poorly managed distributed application environment.

Here are the ETL steps in order, from the resource link below:

Extract the data from the external data sources such as line-of-business systems, CRM systems, relational databases, web services, and anywhere you have business data you want to summarize.

Transform the data. This includes cleansing the data and converting it to an OLAP-friendly data model. The OLAP-friendly data model traditionally consists of dimension and fact tables in a star or snowflake schema.

Load the data so that it can be quickly accessed by querying tools such as reports, dashboards, etc.

"90% of the requirements now is better than 100% of the requirements never" -Unknown

Reference: https://blogs.msdn.microsoft.com/andreasderuiter/2012/12/05/designing-an-etl-process-with-ssis-two-approaches-to-extracting-and-transforming-data/

SOLID

"These principles, when combined together, make it easy for a programmer to develop software that are easy to maintain and extend." -Robert 'Uncle Bob' Martin

S Single-Responsibility - objects should only serve one purpose

O Open-Closed - types should be open to extending capabilities, closed to base changes

L Liskov Substitution - subtypes of base types should produce objects compatible with base

I Interface Segregation - client types do not have to depend on unused interface methods

DDependency Inversion - rely on abstract templates, not prescriptive/restrictive concretes



Reference: https://scotch.io/bar-talk/s-o-l-i-d-the-first-five-principles-of-object-oriented-design

Levity on the Never-ending Race Towards the Next Big IT Trend

Because, c'mon now- at some level, every one of us can relate to 1 or both of these guys 😆

Log Shipping and Log (and/or Differential Backup) Restoration

Most developers know what a full database backup is. Did you know a differential backup is just a subset of the full backup? But what is the point of transaction log shipping in SQL Server?

A "transaction log" is just a subset of a subset of a full backup containing a set of all T-SQL transactions for a given (usually short span) amount of time; they are typically set at intervals of every few minutes but this varies according to application and data compliance needs.

A commenter in the referenced web link below aptly notes:

"the point of log shipping is to keep the secondary server closer to 100% synced than the full backups (which are much larger and produced much less frequently than log backups)."

Log shipping can be used to serve as a constant "warm backup" (similar to SQL Server database mirroring) of all production transactions for a standby/failover SQL Server* as well as a dedicated Backup SQL Server as depicted here:

 Setup if you have a fully-functioning Secondary Production SQL Server

Or, more commonly, the shipped backup logs are sent to a single Backup Server:

Typical production database server scenario, though the steps are a bit out of sequential order, if not (or due to) visual order

In short, a full backup contains the entire database you to recover- up to a certain point in time.

A differential backup is a subset of a backup (generally "full" backups contain days worth of data for bigger organizations)- it would only be prudent to include one or even four differential backups to ensure quick disaster recovery (and full recovery of vital customer data).

Backup logs are chunks of SQL transactions that eventually compose the differential backup(s) and finally the entire full .bak file.

These 3 disaster recovery tools that SQL Server affords DBAs and developers allow you to restore your production data back to the point in time when your server failed:

  • Apply the most recent full backup of the prod server that went down*
  • Apply any differential backups since the most recent full backup
  • Apply all transaction logs up to the point in time of the failure**
  • Apply all StandBy server transactions logs that may have occurred (if not ReadIOnly)

With all of this said, do be aware that different scenarios and requirements call for different kind of disaster recovery solutions. Consult with the experts and never neglect to TEST YOUR RESTORE PROCEDURE WITH ACTUAL .bak and .trn files- and then test the applications and services connected to those database servers against their restored/Secondary version to ensure you suffer as little downtime as possible in the event of a database emergency.

What a restoration timeline might look like

*Note(!) to the questions on Microsoft Licensing and your StandBy server, should it need to be used for any production activity during your disaster recovery:

"I am not the licensing police, and I am not Microsoft – check with your licensing representative to clarify your exact situation. Generally, you can have one warm standby server. However, the second someone starts using it for reporting, testing, or anything else, you need to license it like any other server."-Brent Ozar

**"If the database is in Standby mode, users can query it – except when a log backup is being restored. You need to decide if a restore job will disconnect users, or if the restore is delayed until after the users are disconnected." -Brent Ozar

References:

https://www.sqlshack.com/sql-server-disaster-recovery/ (SQL Server Replication is another compliment or alternative to the log shipping and backup/restore)

 https://www.brentozar.com/archive/2013/03/log-shipping-faq/

Blockchain

Blockchain, apart from all the hoopla heralding it as a "cryptic" means of facilitating monetary transactions, is essentially just a distributed digital ledger (multiple sources of the exact same truth) and distributed digital notaries (transaction verifiers) who, unlike human money handlers and human notaries- do not lie.


Transactions can be scheduled via any desired logic (ie. loan repayment terms, payroll distribution, annuity payouts, securities purchase/sale/options, etc.) and encryption lies at the foundation of the technology- ensuring data integrity and information security.

Blockchain is a type of DLT design. DLT is well-illustrated below:

© 2016 LPEA

Much like Git commits, each blockchain transaction is forever tied to an unmodifiable unique SHA-256 hash sum value that ensures the immutability (un-changeability) of each transaction- in brief, if the transaction changes, the original SHA-256 hash sum (unique file signature) that has already been distributed to all nodes of the blockchain- will not match the modified transaction's SHA-256 hash sum, and everyone connected to the blockchain will be able to see that someone is trying to commit a fraudulent transaction modification. So changes to already-posted transactions within a blockchain digital ledger (in theory at least) are virtually impossible.


Temporal Tables in SQL Server

Definition from Microsoft: SQL Temporal Tables, also known as "system-versioned tables" are a database feature that brings built-in support for providing information about data stored in the table at any point in time rather than only the data that is correct at the current moment in time. Temporal is a database feature that was introduced in ANSI SQL 2011.

What is it used for?
1). Auditing Data History- identifying who did what, when, and row-level restoration to the state of the temporal table data any point in time. (Time travel)

2). Monitor Data Deltas (ie. maintain company sales discounting policy by ensuring no product has had a price change that exceeds some company threshold (200% increase or decrease, etc.))

3). Slowly Changing Dimension management (eliminate the need for complicated ETL logic for tracking SCD- SQL Temporal Tables do this audit tracking automatically)




In Practice
Once upon a time, I worked on a project that had a "unique" (kind euphemism) legacy database schema design. This company had a custom "system" for tracking changes in data. There was no referential integrity so everything relied on application code not having any data access code bugs, which as we all know is not a particularly reliable setup.

Temporal Tables would have done the trick without all the complexity of having to manage a proprietary data versioning system that sometimes worked, oftentimes caused headaches and mass confusion over the root cause of production issues related to versioning of certain data entities.

Temporal Tables have the current or base table, as well as a history table that logs changes over time

Unless there is a specific reason that SQL Temporal Tables won't work for your scenario* you should take advantage of existing functionality like this so that you can focus on the unique functionality of your app and not waste precious time reinventing wheels.

Creating a temporal table is relatively straightforward. In SSMS (connected to an instance of SQL Server 2016 or later), select the option as seen below.

Menu option in SSMS


Here is the T-SQL for basic Temporal Table DDL:

 USE [KpItSimpl]  
 GO  
 --Create system-versioned temporal table. It must have primary key and two datetime2 columns that are part of SYSTEM_TIME period definition  
 CREATE TABLE dbo.SimplTemporalTable  
 (  
      --Table PK and columns definition  
   ProductID int NOT NULL PRIMARY KEY CLUSTERED,  
   ProductDescription varchar(max) NULL,  
      ProductPrice decimal,  
   --Period columns and PERIOD FOR SYSTEM_TIME definition  
   ValidFrom datetime2 (2) GENERATED ALWAYS AS ROW START,   
   ValidTo datetime2 (2) GENERATED ALWAYS AS ROW END,  
   PERIOD FOR SYSTEM_TIME (ValidFrom, ValidTo)   
 )  
  WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = dbo.SimplTemporalTableHistory));   
 GO  
T-SQL DDL for creating a new Temporal Table


For a practical example, I inserted some data into this temporal table, and then committed some UPDATE statements on that data, changing the ProductPrice value of some of the product records. The query below demonstrates how we can look for records (we are likely most interested in the exact change time) that had a price change exceeding +/- 200%:

Example of a query utilizing the Temporal Table feature to see an excessive product price change

Note(!) one limitation that stands out right away- you cannot drop a temporal table like a standard SQL table. You must turn versioning off on the table, drop the table and drop the history table. So instead of selecting "Delete" from the Table menu options or executing "DROP TABLE", you will want to select "SCRIPT AS >> DROP To" option as seen below:

You must script the deletion/drop of a SQL temporal table

As you can see, SQL Temporal Tables can make managing data changes a lot easier. Certainly much easier than a proprietary change versioning system and much faster than restoring the entire database to inspect the state of data at a particular point in time.


*Temporal Table Considerations and Limitations: https://docs.microsoft.com/en-us/sql/relational-databases/tables/temporal-table-considerations-and-limitations?view=sql-server-2017

Reference: https://docs.microsoft.com/en-us/sql/relational-databases/tables/temporal-table-usage-scenarios?view=sql-server-2017