Social Authentication in ASP.NET Core MVC

A common modern convenience for apps is the ability to choose to authenticate and create an account based on credentials from an existing service that most people have (ie. Google, Twitter, LinkedIn, GitHub, etc.). in this post I will walk through the configuration of social authentication for Google and LinkedIn accounts.

Setting up social authentication in ASP.NET Core is a lot easier than you might think...

Before implementing this feature you will need to register for a Google and a LinkedIn developer account which will then give you access to the 2 values that make the magic of this built-in authentication possible: clientKey and clientSecret.

Important(!): you will also want to register the Callback/Redirect URLs for each social authentication as shown below. This redirect URL below is for my project running on localhost:44396 obviously ("/signin-google" is the path you want to append to your app root for Google and "/signin-linkedin" for LinkedIn):


Once you have this setup, you will be able to wire them up in the Startup.cs ConfigureServices() method of an ASP.NET Core 3 MVC Web Application ie:

  public void ConfigureServices(IServiceCollection services)  
     {  
       services.AddDbContext<ApplicationDbContext>(options =>  
         options.UseSqlServer(  
           Configuration.GetConnectionString("DefaultConnection")));  
       services.AddDefaultIdentity<IdentityUser>(options => options.SignIn.RequireConfirmedAccount = true)  
         .AddEntityFrameworkStores<ApplicationDbContext>();  
       services.AddControllersWithViews();  
       services.AddRazorPages();  
       services.AddAuthentication()  
       .AddGoogle(o =>  
       {  
         o.ClientId = "[YourGoogleClientId]";  
         o.ClientSecret = "[YourGoogleClientSecret]";  
       })  
       .AddLinkedIn(o =>  
       {  
         o.ClientId = "[YourLinkedInClientId]";  
         o.ClientSecret = "[YourLinkedInClientSecret]";  
       });  
       // Add application services.   
       services.AddMvc();  
     }  


Now you just have to:

  • Create your ASP.NET Core 3 MVC Web Application project with Identity options as below
  • Install a couple Nuget packages for Google OAuth2 and Linked OAuth respectively
  • Call "Update-Database" from Nuget Package Manager Console
  • Modify Startup.cs

To implement this feature for your users you only need follow a couple steps when setting up the project (whether ASP.NET MVC or the newer ASP.NET Core MVC) to enable some builtin identify and authentication handling after which you can configure and customize to fit your app's particular custom authentication needs. Follow the images below for proper installation of the required 2 Nuget packages, the db command, etc.

First, make sure you create your ASP.NET Core MVC project with "Individual User Accounts" radio button selection and "Store user accounts in app" dropdown selection as follows:


Change Authentication to use Individual User Accounts- this sets up the required boilerplate template code



You will add this Nuget package for LinkedIn authentication



You will add this Nuget package for Google authentication


Next you will need to create the database objects (ASPNET app auth db and tables) via "Update-Database" Package Manager Console command


Finally, modify Startup.cs ConfigureServices() method as shown in the snippet above. Compile and give it a whirl.

And that is all there is to it. Strangely there is not a lot of documentation on exactly how to do this and what can cause it to not work; I spent over an hour debugging what turned out to be a not-so-obvious issue (in my case I was following instructions that erroneously suggested I inject unneeded services to the services.AddAuthentication() instantiation in the snippet of Startup.cs above).


If your keys and Callback URLs are correct you will be able to authenticate to Google and LinkedIn



Once you have registered your social account, you can then log in with it and you will see the following (notice "Hello!...." and "Logout")


If you are like me and do not want to keep to adding yet more and more credentials to LastPass, your users probably don't either. Implementing Social Authentication is a powerful tool that you can leverage with relative ease in ASP.NET Core 3 MVC Web apps.

If you have questions on implementing this or just need some help and general guidance, feel free to comment or drop me an email at colin@sonrai.io


GitHub: https://github.com/cfitzg/DotNetSocialAuth


References:

http://codereform.com/blog/post/asp-net-core-2-1-authentication-with-social-logins/

https://docs.microsoft.com/en-us/aspnet/mvc/overview/security/create-an-aspnet-mvc-5-app-with-facebook-and-google-oauth2-and-openid-sign-on

https://ankitsharmablogs.com/authentication-using-linkedin-asp-net-core-2-0/  

https://stackoverflow.com/questions/53654020/how-to-implement-google-login-in-net-core-without-an-entityframework-provider


Compound Interest

Ultimately the two most important factors in growing money is the size (%) of growth (rate of return or ROR) and the rate of growth (or the number of times the investment earns interest on a certain balance total, that interest is added to the balance, and then the investor begins earning interest on this new, higher amount).

This is why investing in something that will return 5% in 1 year is much worse than an investment that will return 5% quarterly or 4 times a year as illustrated below:

Year 1: $1,000 (5%) == $1,050
-vs- 
Quarter 1: $1,000 (5%) == $1,050.00
Quarter 2: $1,050 (5%) == 1,103
Quarter 3: 1,103 (5%) == $1,158
Quarter 4: $1,158 (5%) == $1,216

Would you rather receive $1,050 or $1,216 dollars after one year of making a $1,000 dollar loan? 🤔

The power of compound interest is even better illustrated in a chart of growth over a period of several years. The below chart illustrates earning 10% per year on a $1,000 investment for 10 consecutive years.

An example with more money and more growth shows more clearly the power of compound interest


The below graphic shows clearly how important it is to save early. In it, Amy is able to earn enough in 10 years of savings, to earn more over the course of 34 years by simply earning interest off the savings of her initial 10 years of investing. No more contributions- just by earning interest she will have earned more than someone else who skipped those first 10 years and began investing later in their career.

In fact, Ethan contributes $100/month for 24 years and is not able to earn as much as Amy did contributing $100/month for just 10 years because Amy began putting her money to work much earlier than Ethan.

So 10 years go by, Amy consistently saving, Ethan consistently not saving anything. Now(?) Ethan, deciding he needs to begin saving for retirement is starting out with a much lower investment ($1,239.72) than Amy is already at by year 10 ($17,485.70).

The true power of compound interest is in the utilization of time and the time value of money. Because money that is invested wisely (presumably) always has a positive return, if you forgo savings (as I and many of my generation have) earlier on in your career you are incurring a huge opportunity cost; that opportunity being 10 years of 10% growth on $100/month investment with compounding interest.

Ethan chose to decline that opportunity and paid for it by losing out on all of that investment + compounded interest he could have received in the first 10 years of his career ($17,485.70), like Amy did.

Simply put, if you plan to invest (in safe, sound investments), the earlier you start the better off you will be. You cannot buy back time and time is a key ingredient to the growth of money.


Amy (wisely) began investing much earlier than Ethan and because of it, she is able to invest far less and still earn more.

Neat UI Effects with jQueryUI and Animate.CSS

So you want neat animation effects on your UI? Seems like it might be a tricky implementation, right?

Animate.css makes it easy to implement modern UI animation effects that make for better user experience/cues


It could not be simpler with jQuery UI and other open source options. Here is an easy sample to pick up and run with from jQuery.com:

jQuery UI Effects
 <!doctype html>  
 <html lang="en">  
 <head>  
   <meta charset="utf-8">  
   <meta name="viewport" content="width=device-width, initial-scale=1">  
   <title>jQuery UI Effects - Effect demo</title>  
   <link rel="stylesheet" href="//code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css">  
   <link rel="stylesheet" href="/resources/demos/style.css">  
   <style>  
     .toggler {  
       width: 500px;  
       height: 200px;  
       position: relative;  
     }  
     #button {  
       padding: .5em 1em;  
       text-decoration: none;  
     }  
     #effect {  
       width: 240px;  
       height: 170px;  
       padding: 0.4em;  
       position: relative;  
     }  
       #effect h3 {  
         margin: 0;  
         padding: 0.4em;  
         text-align: center;  
       }  
     .ui-effects-transfer {  
       border: 2px dotted gray;  
     }  
   </style>  
   <script src="https://code.jquery.com/jquery-1.12.4.js"></script>  
   <script src="https://code.jquery.com/ui/1.12.1/jquery-ui.js"></script>  
   <script>  
     $(function () {  
       // run the currently selected effect  
       function runEffect() {  
         // get effect type from  
         var selectedEffect = $("#effectTypes").val();  
         // Most effect types need no options passed by default  
         var options = {};  
         // some effects have required parameters  
         if (selectedEffect === "scale") {  
           options = { percent: 50 };  
         } else if (selectedEffect === "transfer") {  
           options = { to: "#button", className: "ui-effects-transfer" };  
         } else if (selectedEffect === "size") {  
           options = { to: { width: 200, height: 60 } };  
         }  
         // Run the effect  
         $("#effect").effect(selectedEffect, options, 500, callback);  
       };  
       // Callback function to bring a hidden box back  
       function callback() {  
         setTimeout(function () {  
           $("#effect").removeAttr("style").hide().fadeIn();  
         }, 1000);  
       };  
       // Set effect from select menu value  
       $("#button").on("click", function () {  
         runEffect();  
         return false;  
       });  
     });  
   </script>  
 </head>  
 <body>  
   <div class="toggler" style="margin-top:200px;">  
     <div id="effect" class="ui-widget-content ui-corner-all">  
       <h3 class="ui-widget-header ui-corner-all">Effect</h3>  
       <p>  
         Etiam libero neque, luctus a, eleifend nec, semper at, lorem. Sed pede. Nulla lorem metus, adipiscing ut, luctus sed, hendrerit vitae, mi.  
       </p>  
     </div>  
   </div>  
   <select name="effects" id="effectTypes">  
     <option value="blind">Blind</option>  
     <option value="bounce">Bounce</option>  
     <option value="clip">Clip</option>  
     <option value="drop">Drop</option>  
     <option value="explode">Explode</option>  
     <option value="fade">Fade</option>  
     <option value="fold">Fold</option>  
     <option value="highlight">Highlight</option>  
     <option value="puff">Puff</option>  
     <option value="pulsate">Pulsate</option>  
     <option value="scale">Scale</option>  
     <option value="shake">Shake</option>  
     <option value="size">Size</option>  
     <option value="slide">Slide</option>  
     <option value="transfer">Transfer</option>  
   </select>  
   <button id="button" class="ui-state-default ui-corner-all">Run Effect</button>  
 </body>  
 </html>  


And one I whipped up from Animate.css...:
 <html>  
 <head>  
  <script src="https://code.jquery.com/jquery-3.4.1.min.js" integrity="sha256-CSXorXvZcTkaix6Yvo6HppcZGetbYMGWSFlBw8HfCJo=" crossorigin="anonymous"></script>  
  <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/animate.css/3.7.2/animate.min.css">  
  <script>                 
      $( document ).ready(function() {  
           myDiv = document.querySelector('#myDiv')       
           $("#buttonBounce").on("click", function () {  
                removeAllClasses();  
                myDiv.classList.add('animated', 'bounce')  
           });  
           $("#buttonSwing").on("click", function () {  
                removeAllClasses();  
                myDiv.classList.add('animated', 'swing')  
           });            
           $("#buttonWobble").on("click", function () {  
                removeAllClasses();  
                myDiv.classList.add('animated', 'wobble')            
           });            
           $("#buttonTada").on("click", function () {  
                removeAllClasses();  
                myDiv.classList.add('animated', 'tada')  
           });       
           $("#buttonBounceOut").on("click", function () {  
                removeAllClasses();  
                myDiv.classList.add('animated', 'bounceOutLeft')  
           });            
           function removeAllClasses(){  
                myDiv.classList.remove('animated', 'bounce', 'wobble', 'bounceOutLeft', 'tada', 'swing');  
           }  
      });       
  </script>  
 </head>  
 <body style="background:#000000;">  
 <div id="myDiv" style="margin-top:100px; margin-left:300px; width:250px; background:aqua; font-weight:bold; font-family:Arial; border: 2px solid lime; padding: 10px; border-radius: 25px;">~Something to animate~</h1>  
 <br/><br/>  
 &nbsp;&nbsp;<input type="button" id="buttonBounce" class="ui-state-default ui-corner-all" value="Bounce" />  
 </br><br/>  
 &nbsp;&nbsp;<input type="button" id="buttonTada" class="ui-state-default ui-corner-all" value="Tada" />  
 </br><br/>  
 &nbsp;&nbsp;<input type="button" id="buttonSwing" class="ui-state-default ui-corner-all" value="Swing" />  
 </br><br/>  
 &nbsp;&nbsp;<input type="button" id="buttonWobble" class="ui-state-default ui-corner-all" value="Wobble" />  
 </br><br/>  
 &nbsp;&nbsp;<input type="button" id="buttonBounceOut" class="ui-state-default ui-corner-all" value="Bounce Out" />  
 </br><br/>  
 </body>  
 </html>  


This is what the Animate.css code above will look like..


Happy coding!


Codepens: 

https://codepen.io/radagast27/pen/pooavXd

https://codepen.io/radagast27/pen/xxxYboM



References:

https://jqueryui.com/effect

https://github.com/daneden/animate.css

ASCII Art with Python

Have you ever wanted to convert an icon or photo to ASCII art? Fortunately there are lots of tools to do this out there. I looked around for one that works with Python 3+ and found a succinct, modular script that can output ASCII art from image files like the following:


The script is from Paul Bourke. There is very clear commenting on each step. Essentially, an image loaded from the "--file" argument is divided into a grid and each position of the grid is assigned certain ASCII character whose gray scale level most closely matches the average for that area of the image. I simply modified line 146 and 150 to demonstrate on console vs. the outfile.txt default.

Save the following as ascii.py:
 #Python code to convert an image to ASCII image.  
 import sys, random, argparse  
 import numpy as np  
 import math  
 from PIL import Image  
 # gray scale level values from:  
 # http://paulbourke.net/dataformats/asciiart/  
 # 70 levels of gray  
 gscale1 = "$@B%8&WM#*oahkbdpqwmZO0QLCJUYXzcvunxrjft/\|()1{}[]?-_+~<>i!lI;:,\"^`'. "   
 # 10 levels of gray  
 gscale2 = '@%#*+=-:. '   
 def getAverageL(image):  
      # get image as numpy array  
      im = np.array(image)  
      # get shape  
      w,h = im.shape  
      # get average  
      return np.average(im.reshape(w*h))  
 def covertImageToAscii(fileName, cols, scale, moreLevels):  
      # declare globals  
      global gscale1, gscale2  
      # open image and convert to grayscale  
      image = Image.open(fileName).convert('L')  
      # store dimensions  
      W, H = image.size[0], image.size[1]  
      print("input image dims: %d x %d" % (W, H))  
      # compute width of tile  
      w = W/cols  
      # compute tile height based on aspect ratio and scale  
      h = w/scale  
      # compute number of rows  
      rows = int(H/h)  
      print("cols: %d, rows: %d" % (cols, rows))  
      print("tile dims: %d x %d" % (w, h))  
      # check if image size is too small  
      if cols > W or rows > H:  
           print("Image too small for specified cols!")  
           exit(0)  
      # ascii image is a list of character strings  
      aimg = []  
      # generate list of dimensions  
      for j in range(rows):  
           y1 = int(j*h)  
           y2 = int((j+1)*h)  
           # correct last tile  
           if j == rows-1:  
                y2 = H  
           # append an empty string  
           aimg.append("")  
           for i in range(cols):  
                # crop image to tile  
                x1 = int(i*w)  
                x2 = int((i+1)*w)  
                # correct last tile  
                if i == cols-1:  
                     x2 = W  
                # crop image to extract tile  
                img = image.crop((x1, y1, x2, y2))  
                # get average luminance  
                avg = int(getAverageL(img))  
                # look up ascii char  
                if moreLevels:  
                     gsval = gscale1[int((avg*69)/255)]  
                else:  
                     gsval = gscale2[int((avg*9)/255)]  
                # append ascii char to string  
                aimg[j] += gsval  
      # return txt image  
      return aimg  
 # main() function  
 def main():  
      # create parser  
      descStr = "This program converts an image into ASCII art."   
      parser = argparse.ArgumentParser(description=descStr)  
      # add expected arguments  
      parser.add_argument('--file', dest='imgFile', required=True)  
      parser.add_argument('--scale', dest='scale', required=False)  
      parser.add_argument('--out', dest='outFile', required=False)  
      parser.add_argument('--cols', dest='cols', required=False)  
      parser.add_argument('--morelevels',dest='moreLevels',action='store_true')  
      # parse args  
      args = parser.parse_args()  
      imgFile = args.imgFile  
      # set output file  
      outFile = 'out.txt'   
      if args.outFile:  
           outFile = args.outFile  
      # set scale default as 0.43 which suits  
      # a Courier font  
      scale = 0.43   
      if args.scale:  
           scale = float(args.scale)  
      # set cols  
      cols = 80   
      if args.cols:  
           cols = int(args.cols)  
      print('generating ASCII art...')  
      # convert image to ascii txt  
      aimg = covertImageToAscii(imgFile, cols, scale, args.moreLevels)  
      # open file  
      f = open(outFile, 'w')  
      # write to <console or> file ..f.write(row + '\n')  
      for row in aimg:  
           print(row + '\n')   
      # cleanup  
      f.close()  
      print("ASCII art written to console...") # %s" % outFile)  
 # call main  
 if __name__ == '__main__':  
      main()  


Using Windows CMD, navigate to the directory where your ascii.py script is located. Then execute the following:

ascii.py --file C:\somedir\logo.jpg --cols 150


Referencehttp://paulbourke.net/dataformats/asciiart/

DevOps as Culture

DevOps is an emerging IT role but also a culture; and changing culture does not happen overnight

  • Elevate empowerment and give courage to people to "raise hand" and volunteer to fix things. Even if they are wrong- interest in areas outside one's primary role is good for the entire DevOps culture.
  • Shared accountability - no blame game between developers, PMs, testers, QA, etc.
  • Do not want to have to be dependent on "lone genius" or "firefighter"- need to share and transfer knowledge.
  • Offer time to learn. Encourage hacking for new features and for hardening security and find bugs.
  • Embrace failures in retrospectives to prevent repeated mistakes on future work.
  • Provide the right incentives to motivatve the values you want to reward: reward delivery of quality vs. fire fighting.
  • Understand "value streams" (esp. value stream bottleneck and how can we optimize all constraints) to know where to spend time accordingly.
  • Focus on CONSTRAINTS.
  • Avoid Configuration Drift- Config changes should cascade to all environments (QA, DEV, TEST, STAGE, PROD).
  • Automate the Path to Production.
  • Use pull-based systems so that people integrate each others changes and learn how everything works in unison/concert.
  • People should not fear for their jobs- Systems Admin becomes more important not less so, in DevOps
  • DevOps is how you work not just what you buy or what tech you are using.
  • Total adoption happens in stages/iterations.
  • Traditional Project and Documentation mindset is outmoded, outdated, and disconnected from a living IT mission.
  • Lack of DevOps leads to waste and waiting (waiting for ppl with the right skills to work on things vs. having team that can easily shift contexts or frameworks/languages).
  • SHARE KNOWLEDGE AND EMPOWER COLLEAGUES TO DO MULTIPLE TASKS AND UNDERSTAND SOME OR ALL ASPECTS OF MULTIPLE RESPONSIBILITIES OF THE SOFTWARE PROCESS CHAIN.


Why NoSQL is Never Going to Replace SQL (apple:orange)

Do you need a jet engine to get your results from point A to B or will a modest GM sedan suffice?

I'll take the above bad analogy further and posit that while sedans and cars on the ground require stringent rules and have to navigate much more rigid structures, a jet engine simply powers the jet ahead through constraint-less skies- it's purpose is to power something big- not to be concerned with other machinery of the craft (ie. RDBMS features eschewed by NoSQL solutions).

Do you need massive global data sync scale so that millions can connect and make changes and the results all (appear) real-time? If not, NoSQL is not always the right choice and neglecting to have any kind of schema for stored application data structures can present its own host of challenges in the future if (when) those structures change. But alas, you can use NoSQL for some things (Redis, image/BLOB storage) and an RDBMS for others (more structured records and things you want to restore to a point in time in the event of a server failure).

Relational databases tend to be scaled up; NoSQL solutions are scaled out

The SQL vs NoSQL (structured and transactional vs. semi-structured and "eventually consistent") debate is not a matter of one or the other and that's that. These are complementary technologies and should both be used- wherever you find app requirements that suggest one or the other makes the most sense.

When an application is meant to scale immensely and there is not a lot of data integrity, consistency, transaction or complex data structuring and transformation needs- NoSQL is your best bet and will far outscale even the most robust RDBMS server farm- at least at a much lower cost (at the cost of sacrificing features of an RDBMS which may not be needed).

I have personally worked on several projects that utilize relational and unstructured approaches to reading and persisting application data. If you have ever used an application's config file to change a setting in JSON or XML or a simple line entry- you are seeing a small and very basic NoSQL example of storing app data.

Hadoop and other distributed NoSQL db servers are built for scalability


Using NoSQL in software development can make data structures and objects- passed to and fro from APIs and within the application itself- much more flexible to work with. 1-line to serialize an object to JSON chunk, save it to BLOB storage and forget about it.

When dealing with relational data, you really have to understand the data to write good data access code and the underlying SQL that supports well-defined structuring of complex objects.

Well-defined structuring of the persistence of complex application objects avoids data duplication/corruption, prevents breaking reference constraints and losing any sense of hierarchical data relationships and more generally lets you know very quickly when you have a problem within your data storage structures and the objects that initialize themselves from that data.

NoSQL ditches virtually all relational database data normalization rules in favor of a loosely schema'd unstructured (document, BLOB, KeyStore, etc.) data store that relies solely on keys, values and filtering unstructured metadata to get the same SELECT ... WHERE functionality found in RDBMS. Its iterations usually bear a resemblance to Java and as loosely follows here are common SQL statements and their Java or Java-derived equivalent:


"Apache Hadoop is an open source platform built on two technologies: Linux operating system and Java programming language."


For many application data requirements however, relational data can be overkill and totally unnecessary (ie. Redis to store key/vals vs. designing some elaborate key/val store in a SQL Server table).


Implementing something like Splunk or Kibana to continously index app logs and configuration files can help you dip toes into the Lake of Dark Data


The best distributed NoSQL solutions like Hadoop really shine in their inherit ability to dynamically scale to as many server machines as the operators can make ready to serve as "Hadoop processor nodes on standby".

SQL Server scaling is based more on server augmenting or "scaling up" (adding RAM, faster SSDs, RAID Arrays, etc.) rather than distributing workloads across dynamic nodes. SQL Server AlwaysOn Availability and its Mirroring and Replication feature are for recoverability and data sharing- not dynamic scaling to handle bigger and bigger workloads.

SQL has been around forever. The fundamental concept behind NoSQL (semi-structured or loosely structured data) has been around since long before SQL relational database technology. Both (along with NoSQL-related graph database paradigm) will continue to serve as viable data storage solution alternatives for many more years into the 21st century.


Relational systems like SQL Server, MySQL and Oracle usually handle structured side; NoSQL vendors are after the other 90%


In fact, SQL Server 2019's Polybase extension supports Hadoop Clusters, MongoDB and Terradata T-SQL query integration. A new feature called SQL Server Big Data Clusters helps make distributed NoSQL nodes manageable within SSMS environment.

Mongo, Hadoop and other NoSQL database servers have SQL server integration to support relational data sources.

SQL Server 2019 Polybase integrates Hadoop, MongoDB and many other sources with relational data and T-SQL queries


CAP Theorem: a distributed data system like most all NoSQL solutions can only achieve 2 of the 3 features: "Consistency", "Availability" and "Partition Tolerance"

ACID vs BASE:  The relational axiom of "Atomic, Consistent, Isolated, Durable" contrasted against NoSQL's vague promise of "Basically Availability, Soft State, Eventual Consistency" (dirty reads common)

This ol' tried-and-true database server software ain't going away in the foreseeable future


Hundreds of millions of corporate, mid and small business applications are running along just fine in 2019 using various RDBMS platforms (SQL Server, Oracle, DB2, PostgreSQL, MySQL, etc.) for at least one of their data stores.

Many more millions of applications have been using one riff or another of NoSQL (semi-structured data) before, during and after the mythical "Relational Movement" as described by software veteran Robin Bloor:

"The Relational Model of Data Never Dominated Anyway. Estimates vary, but it is generally agreed that somewhere between 70% and 95% of the world’s data is stored only in poorly structured or unstructured formats such as: word processing documents, spreadsheets, HTML files and e-mail. The truth is that Relational database never did really dominate. It was rejected out of hand, year after year, as an effective store for many types of data." -Robin Bloor on insideanalysis.com


Google search trends over the last 5yrs certainly suggest relational SQL is not going anywhere anytime soon...


Considerations when evaluating whether to use NoSQL:
  • NoSQL is a precise tool for precise data needs; if relational SQL is too much for your group, NoSQL will likely be too steep a learning curve
  • Data Integrity- when billions of NoSQL records are affected by a small change in schema that is not able to propagate correctly or runs into constraint issues or hierachy and relations are impossible to infer... maybe relational SQL would be a better approach
  • NoSQL touts loose schema structure is a benefit but this simply means schema and data structure enforcement has been shifted from the database layer to the application layer. Data cannot "self-manage".
  • Some apps are prime candidates for NoSQL's document-centric and resource-centric distributed storage architecture


Also, there is this to consider:


(re: the longevity and simple-yet-powerful abstractions of SQL)


If NoSQL solutions are eventually able to achieve the same transactional consistency and complex schema structures that some applications require and then ultimately subsume RDBMS completely- it'll still require a lot of SQL gurus to convert and integrate all the legacy relational database apps for a long, long time to come...

Bring on MongoDB, CouchDB, Dynamo, MapReduce, HBase, BigTable, Cassandra.


As data professionals we will have an increasingly complex array of tools to understand; what we do with them will drive the future



Long live SQL Server 2030. 😉




References:

https://blog.timescale.com/why-sql-beating-nosql-what-this-means-for-future-of-data-time-series-database-348b777b847a/

https://www.wired.com/2012/12/couchdb/

https://pdfs.semanticscholar.org/a6f0/1c9103d3bafb8ce92641c9f2a4deaccd12f9.pdf

https://www.memsql.com/blog/why-nosql-databases-wrong-tool-for-modern-application/

https://asktom.oracle.com/pls/apex/f?p=100:11:0::::P11_QUESTION_ID:1589423200346982646

https://news.ycombinator.com/item?id=479165

https://insideanalysis.com/is-the-relational-database-doomed/

NLog for Flexible Application Logging in .NET

As much as I have enjoyed working with log4net in the psat NLog works much more seamlessly for .NET apps (it has good, solid abstractions)

From Nuget Package Manager, you can find and reference NLog and the accompanying NLog.Config which simplifies setup. To configure NLog logging, simply point the required properties (you need at least a logFile variable, a target and a rule to get started) to your desired values in the config file that NLog.Config creates in your project root  (NLog.config):

Configuration:

 <?xml version="1.0" encoding="utf-8" ?>  
 <nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"  
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  
    xsi:schemaLocation="http://www.nlog-project.org/schemas/NLog.xsd NLog.xsd"  
    autoReload="true"  
    throwExceptions="false"  
    internalLogLevel="Off" internalLogFile="c:\temp\nlog-internal.log">  
  <variable name="logFilePath" value="C:\NLog\IDG.log" />  
  <targets>  
   <target name="logfile"  
     xsi:type="File"  
     fileName="${logFilePath}"  
     layout="${longdate}  LEVEL=${level:upperCase=true}: ${message}"  
     keepFileOpen="true" />  
  </targets>  
  <rules>  
   <logger name="*" minlevel="Info" writeTo="logfile" />  
   <logger name="*" minlevel="Warn" writeTo="logFile"/>  
  </rules>  
 </nlog>  
This example uses a log file target in C:\NLog directory; you can utilize a wide variety of logging targets to broadcast app errors

Source Code:

 using Microsoft.VisualStudio.TestTools.UnitTesting;  
 using NLog;  
 namespace ExtRSTests  
 {  
   [TestClass]  
   public class NLogTests  
   {  
     private static Logger logger = LogManager.GetCurrentClassLogger();  
     [TestMethod]  
     public void TestLoggerToFile()  
     {  
       logger.Warn("Something in the app happened that may indicate trouble ahead....");  
       logger.Error("Uh-oh. Something broke.");  
     }  
   }  
 }  
With the logging levels, log formatting (timestamps) and abundant integration options, NLog is a complete logging solution

With NLogger you can implement logging for an array of targets including file, email, database and 3rd party integrations (ie. send message to Slack channel if logger generates any "Error" or "Fatal" level log message).


Reference: https://www.infoworld.com/article/3144535/how-to-work-with-nlog-in-net.html

.NET XML Serialization with System.Xml.Serialization

From SOAP, to app configuration, model templating, EDI/ETL, and beyond, XML is a mainstay in software and will continue to be (see- HTML)


Source Code:
 using System;  
 using Microsoft.VisualStudio.TestTools.UnitTesting;  
 using System.Xml.Serialization;  
 using System.IO;  
 using System.Xml.Linq;  
 using System.Xml;  
 namespace DemoTests  
 {  
   [TestClass]  
   public class TextXmlSerialization  
   {  
     public const string xml =  
       "<XMLEntityCollection>" +  
         "<XMLEntities>" +  
           "<XMLEntity>" +   
             "<ID>1</ID>" +   
             "<Name>TestName1</Name>" +   
           "</XMLEntity>" +  
           "<XMLEntity>" +  
             "<ID>2</ID>" +  
             "<Name>TestName2</Name>" +  
           "</XMLEntity>" +  
         "</XMLEntities>" +  
       "</XMLEntityCollection>";  
     [TestMethod]  
     public void TestDeserialize()  
     {  
       XmlSerializer serializer = new XmlSerializer(typeof(XMLEntityCollection));  
       using (StringReader reader = new StringReader(xml))  
       {  
         XMLEntityCollection ents = (XMLEntityCollection)serializer.Deserialize(reader);  
         TestSerialize(ents);  
       }  
     }  
     [TestMethod]  
     public void TestSerialize(XMLEntityCollection ents)  
     {  
       XmlSerializer xmlSerializer = new XmlSerializer(ents.GetType());  
       using (StringWriter stringWriter = new StringWriter())  
       {  
         xmlSerializer.Serialize(stringWriter, ents);  
         Assert.IsNotNull(stringWriter);  
       }  
     }  
   }  
   [Serializable()]  
   public class XMLEntity  
   {  
     [XmlElement("ID")]  
     public string Id { get; set; }  
     [XmlElement("Name")]  
     public string Name { get; set; }  
   }  
   [Serializable()]  
   [XmlRoot("XMLEntityCollection")]  
   public class XMLEntityCollection  
   {  
     [XmlArray("XMLEntities")]  
     [XmlArrayItem("XMLEntity", typeof(XMLEntity))]  
     public XMLEntity[] Ents { get; set; }  
   }  
 }  



.NET XML Deserialization with System.Xml.Serialization: 


Instantiated (ents) object containing XML from the deserialized XML string



.NET XML Serialization with System.Xml.Serialization:


XML genertaed  by the stringWriter when serializing ents object into XML using the NET XMLSerializer type



Reference: https://stackoverflow.com/questions/364253/how-to-deserialize-xml-document

Newtonsoft JSON .NET Nuget Nugget

Most API payloads are in XML or JSON; it is best to know both of these data structures, and how to serialize/deserialize them

The JSON parsing utilities found in Newtonsoft.Json are very. very useful and should be common knowledge for any .NET developer who works with API data or anything producing or derived from JSON (JavaScript Object Notation).

In general, to use Newtonsoft.Json you simply need to create a .NET class hierarchy that mimics the structure and hierarchy of the target JSON. Once that is setup, serializing in-memory objects to JSON and deserializing the JSON back to in-memory objects is a breeze.

You achieve this by normal class hierarchy and making List<> of child objects, array properties, etc. Newtonsoft's 'JsonProperty' class property decorator maps JSON properties and the builtin serialization and deserialization methods facilitate working between JSON strings and the in-memory objects they represent.

The C# source code below demonstrates serialization from a SQL Server 2017 SSRS API v2 JSON response and then serializing that object back into JSON.

Source Code:

 using System;  
 using Microsoft.VisualStudio.TestTools.UnitTesting;  
 using Newtonsoft.Json;  
 using System.Collections.Generic;  
 using System.Net.Http;  
 using System.Threading.Tasks;  
 namespace DemoTests  
 {  
   [TestClass]  
   public class DemoTestJSON  
   {  
     [TestMethod]  
     public async Task TestDeserializeJSON()  
     {  
       HttpClient client = new HttpClient(new HttpClientHandler() { UseDefaultCredentials = true });  
       client.BaseAddress = new Uri("http://localhost/reports/api/v2.0/reports");  
       var response = await client.GetAsync(client.BaseAddress);  
       var deserial = JsonConvert.DeserializeObject<APIGenericItemsResponse>(await response.Content.ReadAsStringAsync());       
       TestSerializeJSON(deserial);  
       Assert.IsNotNull(deserial);  
     }  
     [TestMethod]  
     public void TestSerializeJSON(APIGenericItemsResponse genericObject)  
     {      
       string serial = JsonConvert.SerializeObject(genericObject);  
       Assert.IsNotNull(null);  
     }  
   }  
   public class APIGenericItemsResponse  
   {  
     [JsonProperty("@odata.context")]  
     public string Context { get; set; }  
     [JsonProperty("value")]  
     public List<GenericItem> GenericItem { get; set; }  
   }  
   public class GenericItem  
   {  
     [JsonProperty("Id")]  
     public string Id { get; set; }  
     [JsonProperty("Name")]  
     public string Name { get; set; }  
     [JsonProperty("Path")]  
     public string Path { get; set; }  
   }  
 }  


SSRS API v2 /Reports JSON Response:




JSON Deserialization with Newtonsoft.Json:
The deserial variable holds an in-memory .NET object of type APIGenericResponse, derived (deserialized) from the SSRS API JSON response




JSON Serialization with Newtonsoft.Json:

The serial variable is simply the serialization APIGenericItemsResponse object serialized into a JSON string


Reference: https://www.newtonsoft.com/json/help/html/Introduction.htm

base in C#

The base keyword in C# allows a subclass to access base (superclass) members.

All credit to Suresh Dasari of Tutlane (reference below) on explaining this so effectively in just a few steps of code.

What is shown here is the Details subclass overriding the Users base class' "GetInfo()" method and including the base behavior (Console.WriteLine("Name: {0}", name); ... Console.WriteLine("Location: {0}", location); - along with- some new behavior (Console.WriteLine("Age: {0}", base.age);). In this way members can be shared between subtypes and the type they inherit from- in constructors as well as elsewhere in the subclass.

 using System;  
 namespace Tutlane  
 {  
   // Base Class  
   public class Users  
   {  
     public string name = "Suresh Dasari";  
     public string location = "Hyderabad";  
     public int age = 32;  
     public virtual void GetInfo()  
     {  
       Console.WriteLine("Name: {0}", name);  
       Console.WriteLine("Location: {0}", location);  
     }  
   }  
   // Derived Class  
   public class Details : Users  
   {  
     public override void GetInfo()  
     {  
       base.GetInfo();  
       Console.WriteLine("Age: {0}", base.age);  
     }  
   }  
   class Program  
   {  
     static void Main(string[] args)  
     {  
       Details d = new Details();  
       d.GetInfo();  
       Console.WriteLine("\nPress Enter Key to Exit..");  
       Console.ReadLine();  
     }  
   }  
 }  


Result

Reference: https://www.tutlane.com/tutorial/csharp/csharp-base-keyword

IP Addressing and Subnets, Subnet Masking

Knowledge of network configuration and administration is an (incredibly- still) underrated, underappreciated and immensely powerful tool for any IT professional to possess.

All subnet masking schemes, the mask bits in binary, available number of hosts. A "/24" is common for small LAN subnets.


One area of computer networking that should be more well-understood by software developers is the configuration of subnetworks via subnet masks. A subnet mask (ie. 255.255.255.0) is simply a way of re-purposing an IP Address by segmenting it into network and host portions.

An IPv4 address consists of 4 bytes (32 bits) of data. Each of those bytes contain 8 bits known as "octets". In a 255.255.255.0 subnet mask- all but the last octet is being used for the network ID portion of the IP address and so are ignored.

At this point we could get into the logical ANDing of IP address bits and subnet mask bits but just be aware that the masking bits allow for the network portion of the IP address to be separated from the host portion- that is they key purpose of subnetting and the subnet mask.

The breakdown of a Class B IPv4 address

The subnet mask is designed to denote the number of bits in an IP address (ie. 10.9.1.14) that form the network portion (10.9.1) vs. the host portion (.14).

In this way, IPs can be used in ways they were not originally designed- but that are altogether needed for proper organization of something that has grown as seemingly unwieldy as IP networks of "the Internet" (publically accessible networks of subnetworks). With a little reference knowledge you can understand even the trickiest of subnet configurations.

But wait- there is (lots) more...

The example above illustrates only a very basic subnetting situation.

Where things get tricky is when a subnet mask ends not at the end of an entire octet, but just before the start of the host portion of the IP- in the same octet (ie. 255.255.128.0). In more complex network configuration scenarios it is helpful to refer to a subnet configuration reference sheet like the following to identify the subnet and/or subnet mask information you are looking for:


Describing the nature of a /29 subnet solely from knowing the IP address (10.1.1.37) of one of its hosts and that it is a /29 subnet.


Below are the 7 common pieces of information that you will need to know when analyzing subnet configurations:

Network ID: First available IP address in the subnet.

Broadcast IP: Last available address in the subnet.

First Host IP: Network ID + 1

Last Host IP: Broadcast IP - 1

Next Network: Broadcast  + 1

# of IP Addresses: Number of IP addresses in the subnet range (subtract 2 to find the number of "usable" device IP addresses) - refer to the Subnet Mask Reference Sheet


This enlightening example shows how MCI uses 11 bits of mask, Automation Research Systems 22 bits, ARS 24 bits, freesoft.org 32 bits- all on the same IP address; you can see the subnet hierarchy as MCI controls the entire 208.128.0.0/11 network


Online CIDR Calculator showing MCI subnet breakdown which includes the other 3 subnets shown

IP Points to remember:

  • IP octets (base 10 representation) are 0-inclusive so only ever max of .255 in any given octet.
  • Subnet Mask is a 32-bit number that indicates how many bits of an IP address are used to indicate the network portion vs. host portion and is a way to subdivide networks for organization, security and manageability.
  • The first two available host addresses are network (generally .0), then router (generally .1) and the last available host address (generally .255) is used as the subnet's broadcast address- note these example octets are small LAN defaults/generalities and likely will not apply to a complex subnet.
  • Class A (0-127) uses 8 bits for the network portion of the IP address, leaving 24 bits for host IDs
  • Class B (128-191) uses 16 bits for the network portion of the IP address, leaving 16 bits for host IDs
  • Class C (192-223) uses 24 bits for the network portion of the IP address, leaving 8 bits for host IDs
  • CIDR is the acronym for Classless Inter-domain Routing. It (/26, /24, etc.) is just the number of IP address bits used by the subnet mask (255.255.255.0 = /24 or 24 bits of mask, .255.255.255.192 = /26 or 26 bits of mask).
  • When sorting through IP ranges to determine which range a particular subnetwork group is in, use these time saving tricks recommended by PracticalNetworking:
    • (1) multiply group size by 10 as a (*10) multiple of the group size will be reached
    • (2) if multiplying group size by 10 goes beyond the IP address for which you are trying to find the subnetwork range, remember that "every group size will land on 128 eventually"- so you can use that for a starting basis as well.
    • (3) every group size lands on the subnet value of the selected subnet and every subnet to the left of it (ie. for a /27 subnet or ".224" subnet mask- .224, .192 and .128 will all match the start of a group)

References:

https://www.youtube.com/watch?v=s_Ntt6eTn94&ab_channel=PowerCertAnimatedVideos

https://www.youtube.com/watch?v=BWZ-MHIhqjM

http://www.subnet-calculator.com/

https://www.pcwdld.com/subnet-mask-cheat-sheet-guide

Graphical Integrity

"The representation of numbers, as physically measured on the surface of the graphic itself, should be directly proportional to the quantities represented." -Edward Tufte

It is amazing how easy it is to find highly inaccurate and misleading data graphics and charts even in this year 2019. These inaccuracies and sometimes outright perversions of the truth are of particular concern to an insta-culture who gets its news in headlines, memes, charts and other bite-sized generalizations via social media and rarely looks for the evidence beyond the headlines and the source data behind the charts.

The “Lie Factor”, first defined by American statistician Edward Tufte is defined as "a value to describe the relation between the size of effect shown in a graphic and the size of effect shown in the data." A larger Lie Factor value indicates a higher level of deception or "inaccurate scaling/weighting".


Lie Factor in Action:

The numbers do not equate to the scale of the bars and money bags... not quite as "strong" as projected.




This example mixes 2 different scales and data sets and only serves to confuse the reader...




This is a propaganda data graphic displaying a series of 5 increases using a totally nonsensical scale



This graphic shows Last Year, Last Week, and Current Week as having the same temporal scale.... O'Lie Factor.




Lie Factor Breakdown:

Lie Factor is the change shown in the graphic (say 100%) divided by the change reported in the data (say "50%") - (100/50 = a LF of 2)


There are reasons for misleading graphics that go beyond propaganda and sensationalist news articles:

  • Lack of quantitative skills on the part of the graphic creator and publication editor
  • Doctrine that statistics are boring and therefor need to be "jazzed up"
  • Doctrine that graphics are only for unsophisticated and so don't need "accuracy constraints"
  • Failure to treat graphics with the same fidelity to the truth as the written word it accompanies

Other ways that graphical information displays are corrupted include cherry-picking data, making small changes appear large by showing a small scale interval and when all else fails for information manipulators- using fake data.

It is important to not jump to conclusions when assessing graphical information displays even if it is coming from a reputable publisher. As you can see it is not always obvious that the information being communicated graphically is accurate. Wherever possible, get a look at the source data.

"When we see a chart or diagram, we generally interpret its appearance as a sincere desire on the part of the author to inform. In the face of this sincerity, the misuse of graphical material is a perversion of communication, equivalent to putting up a detour sign that leads to an abyss" - Wainer


References:

https://viz.wtf/

https://infovis-wiki.net/wiki/Lie_Factor


Google Maps API

The Google Maps API is a very powerful tool that is relatively easy to use if you have some JavaScript background. The below screen was created with the code that follows (Google API Key obfuscated).


Basic Maps API example with hard-coded lat/long custom markers


Get an API key here: https://developers.google.com/maps/documentation/javascript/get-api-key

Maps API exposed operations reference: https://developers.google.com/maps/documentation/javascript/tutorial

 <!DOCTYPE html>  
 <html>  
  <head>  
   <title>Custom Markers</title>  
   <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no">  
   <meta charset="utf-8">  
   <style>  
    /* Always set the map height explicitly to define the size of the div  
     * element that contains the map. */  
    #map {  
     height: 100%;  
    }  
    /* Optional: Makes the sample page fill the window. */  
    html, body {  
     height: 100%;  
     margin: 0;  
     padding: 0;  
    }  
   </style>  
  </head>  
  <body>  
  <div>  
  Debug/Inspect...  
  </div>  
   <div id="map"></div>  
   <script>  
    var map;  
    function initMap() {  
     map = new google.maps.Map(  
       document.getElementById('map'),  
       {center: new google.maps.LatLng(43.0589, -88.0988), zoom: 11, mapTypeId: 'hybrid'});  
     var icons = {  
      bucks: {  
       icon: 'bucks.png'  
      },  
      brewers: {  
       icon: 'brewers.png'  
      },  
      panthers: {  
       icon: 'panthers.jpg'  
      }  
     };  
     var features = [  
      {  
       position: new google.maps.LatLng(43.0280, -87.9712),  
       type: 'brewers'  
      }, {  
       position: new google.maps.LatLng(42.9930, -87.9210),  
       type: 'panthers'  
      }, {  
       position: new google.maps.LatLng(43.0000, -87.8379),  
       type: 'bucks'  
      }  
     ];  
     // Create markers.  
     for (var i = 0; i < features.length; i++) {  
      var marker = new google.maps.Marker({  
       position: features[i].position,  
       icon: icons[features[i].type].icon,  
       map: map  
      });  
     };  
           //Must have API with access to the Places API to get marker click details without much manual code  
    }  
   </script>  
   <script async defer src="https://maps.googleapis.com/maps/api/js?key=XXXXXXXXXXXXXXXXXXXX&callback=initMap">  
   </script>  
  </body>  
 </html>