Value vs. Reference Type Memory Allocation

I once in the not-too-distant past had a Developer interview go comically off the rails when I (the interviewee) was posed with the simple software development question:

"Describe the difference between a value type variable and a reference type variable"

This is an elementary Comp Sci 101 question that every developer should be able to understand and answer with unflinching confidence.

This is what value type (int a left) and reference type (object a on the right) memory allocation looks like graphically

The correct answer would point out that value type variables ('primitive' in Java) are stored directly in memory within the Stack while reference type variables contain pointers (id) to another memory location in the Heap (which may in turn contain additional memory references (pointers) that compose the entire object).

And (no I swear really) at one point I knew exactly what the difference was. However with time and as development languages and tools become more and more abstracted away from the underlying behavior of memory during application run-time, my knowledge of this simple-yet-important concept went blank.

I stammered though a meandering non-answer about how value types contain "simple" data types that are known as primitives and how reference types refer to more complex objects. Suffice it to say that it was an embarrassing lack of clarity and awareness of how application memory works and likely torpedoed any chance of an offer.

On value and reference types, the Stack and the Heap and Boxing/Unboxing (converting between val and ref)

This is something we should all know as developers. I hope this helped refresh and/or clarify the concept of variable memory allocation for at least a few people out there.

ETL and EDI Using SSIS

ETL is the process by which you can take (Extract) data from various (usually related) data sources, Transform that data to meet your destination system's needs, and finally Load that transformed data into the destination system data store.

Your table structure will be something along the lines of this basic template:

In a real-world db environment Staging, OLAP, OLTP and other data repos may be on different database servers, this is same db server for demonstration

We will use SQL Server Integration Services (SSIS) and develop the SSIS package within the Visual Studio 2017 IDE.


The first step of the SSIS package load (INSERT) the data into a STAGING area database. This allows us to:

• Store off the intermediate data from all sources into analysis-friendly OLAP datastores
• Perform data integrity checks
• Keep extraction and transformation as two strictly separated steps

We load the data from the various source files (.csv, .xls, .xlsx) into SQL Server database Staging table(s) using SSIS Source and Destination Data Flow Tasks connected with the movable data flow arrows. Once you have connected a source and destination you can go into the Destination Data Flow Task and edit the mappings of which source columns should be written to which destination columns.

Next we perform some transformations. This can be anything from a simple ranking or status/flagging/business prioritization algorithm to data cleansing to data partitioning based on certain criteria; the key is that this Transform step is where we apply T-SQL UPDATEs to transform the data once it has all been aggregated in Staging.

Then we refresh the OLAP destination tables using the same kind of Source and Destination Data Flow Tasks and mappings as used for Staging. The OLAP data is used for data analysis.

Finally, we load the cleansed Staging data into our destination system's OLTP database and email or text message the system owner upon successful completion of the SSIS ETL job (or deliver an error if anything fails). The OLTP data stores live transactions.

Bear in mind that most ETL data-flow step mappings are not a 1:1 match; this is just an e2e demo of SSIS ETL in most basic form

Happy ETL'ing, and be sure to watch out for cases of mysterious symbols/characters from miscellaneous data copied from other programs or from other system environments that were using a Language setting (codepage) which is incompatible with your ETL software. Bad data happens more than you think and as we say, GIGO.

Your end result looks like this (all Green Checkmarks indicates all was successful; I recommend using PaperCut for SMTP testing- super cool and useful product

I would attach or GitHub the source code (and will do so upon request) but SSIS project code has a lot of dependencies and can get quite messy for another to re-use project on even just a 'slightly different' machine.

Having used SSIS' now-deprecated predecessor "DTS" (SQL Server Data Transformation Services) and SSIS for many years I can attest to the fact that the best way to learn this product is by diving right in on your own and begin the creation of sources and destinations and source/destination connection managers and control flow events, and .NET integration, and exception event handlers, etc.

You will likely run into some ambiguous and not well-documented errors when developing in SSIS; but persist in your efforts and you can create a very powerful EDI system with the many capabilities of SSIS and the robust Scheduled ETL jobs that it can create.

References:

https://blogs.msdn.microsoft.com/andreasderuiter/2012/12/05/designing-an-etl-process-with-ssis-two-approaches-to-extracting-and-transforming-data/

https://github.com/ChangemakerStudios/Papercut/releases

Java from .NET and .NET from Java

Java or .NET? Why not both (when it is the only viable path)?

jni4net is a proven interop library for Java and .NET. Two brief examples developed by jni4net below merely require that you to specify the jni4net dependency in the (Visual Studio or Eclipse) project.

Calling Java from .NET

 using java.io;  
 using java.lang;  
 using java.util;  
 using net.sf.jni4net;  
 using net.sf.jni4net.adaptors;  
 namespace helloWorldFromCLR  
 {  
   public class Program  
   {  
     private static void Main()  
     {  
       // create bridge, with default setup  
       // it will lookup jni4net.j.jar next to jni4net.n.dll  
       Bridge.CreateJVM(new BridgeSetup(){Verbose=true});  
       // here you go!  
       java.lang.System.@out.println("Hello Java world!");  
       // OK, simple hello is boring, let's play with Java properties  
       // they are Hashtable realy  
       Properties javaSystemProperties = java.lang.System.getProperties();  
       // let's enumerate all keys.   
       // We use Adapt helper to convert enumeration from java o .NET  
       foreach (java.lang.String key in Adapt.Enumeration(javaSystemProperties.keys()))  
       {  
         java.lang.System.@out.print(key);  
         // this is automatic conversion of CLR string to java.lang.String  
         java.lang.System.@out.print(" : ");  
         // we use the hashtable  
         Object value = javaSystemProperties.get(key);  
         // and this is CLR ToString() redirected to Java toString() method  
         string valueToString = value.ToString();  
         java.lang.System.@out.println(valueToString);  
       }  
       // Java output is really Stream  
       PrintStream stream = java.lang.System.@out;  
       // it implements java.io.Flushable interface  
       Flushable flushable = stream;  
       flushable.flush();  
     }  
   }  
 }  

Calling .NET from Java

 import net.sf.jni4net.Bridge;  
 import java.io.IOException;  
 import java.lang.String;  
 import system.*;  
 import system.Object;  
 import system.io.TextWriter;  
 import system.collections.IDictionary;  
 import system.collections.IEnumerator;  

 public class Program {  
      public static void main(String[] args) throws IOException {  
           // create bridge, with default setup  
           // it will lookup jni4net.n.dll next to jni4net.j.jar   
           Bridge.setVerbose(true);  
           Bridge.init();  
           // here you go!  
           Console.WriteLine("Hello .NET world!\n");  
           // OK, simple hello is boring, let's play with System.Environment  
           // they are Hashtable realy  
           final IDictionary variables = system.Environment.GetEnvironmentVariables();  
           // let's enumerate all keys  
           final IEnumerator keys = variables.getKeys().GetEnumerator();  
           while (keys.MoveNext()) {  
                // there hash table is not generic and returns system.Object  
                // but we know is should be system.String, so we could cast  
                final system.String key = (system.String) keys.getCurrent();  
                Console.Write(key);  
                // this is automatic conversion of JVM string to system.String  
                Console.Write(" : ");  
                // we use the hashtable  
                Object value = variables.getItem(key);  
                // and this is JVM toString() redirected to CLR ToString() method  
                String valueToString = value.toString();  
                Console.WriteLine(valueToString);  
           }  
           // Console output is really TextWriter on stream  
           final TextWriter writer = Console.getOut();  
           writer.Flush();  
      }  
 }  

(verbose commenting by Pavel Savara, a jni4net contributor)

References:

http://zamboch.blogspot.com/2009/10/how-calling-from-net-to-java-works.html

http://zamboch.blogspot.com/2009/11/how-calling-from-java-to-net-works-in.html

https://github.com/jni4net/jni4net/tree/master/content/samples

SQL CLR for .NET in SQL Server

You may find yourself with the need to integrate a .NET method within SQL Server to be called as a function. This usually happens when some relatively complex looping and modifying logic is a requirement of a SQL operation.

SQL is a great data language but it is not the right language for some tasks. Creating a SQL CLR from a .NET assembly may be the best approach to some unique situations (and there is a bonus in that, in many cases you can reuse existing .NET code).

Before creating the CLR object we need a .NET .dll; so first we create a basic .NET assembly compile in Release and copy the path the the compiled .dll:

This is our simple .NET CLR method with which we want to run within the SQL Server query execution engine

SQL CLR provides a way for you to integrate complex .NET methods within SQL Server

Import into SQL Server instance via SSMS*: 

Select New Assembly... 

...and then enter the path to your Release .dll

Create T-SQL function or stored procedure to serve as caller for the function and run it:

From here we can see all of the T-SQL code involved; the 3 SQL Server configuration conditions (shown in the 3 EXEC statements) are required

And that is all there is to it. Only use CLR functions when absolutely necessary as RDBMS's like SQL Server are designed to processes relational data in sets, and not to apply complex business logic on individual rows.

But if there is no other way- SQL CLRs could provide you a solution to your code/logic integration problems.


*Warning and Reference: https://blog.netspi.com/attacking-sql-server-clr-assemblies/

Calling Win32 API from .NET C# Application

As quoted in the useful reference below:

"Anybody doing any serious Windows development in C# will probably end up calling many Win32 functions. The .NET framework just doesn't cover 100% of the Win32 API." - Mike Thompson

This illustrative example here is simply to show what the integrated code looks like; however identifying available drive space is a common app requirement

The interop of .NET and Win32API works by way of referencing the InteropServices .NET namespace and using normal Win32 API functions with the [DllImport()] attribute denoting the Win32 API assembly being used and the corresponding function being modified as "static extern" which informs the compiler that the function is calling unmanaged (non-.NET) code.

 using System;  
 using System.Runtime.InteropServices;  
 namespace ConsoleApp1  
 {  
   internal static class Win32  
   {  
     [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]  
     internal static extern bool GetDiskFreeSpaceEx(string drive, out long freeBytesForUser, out long totalBytes, out long freeBytes);  
   }  
   class Program  
   {  
     static void Main(string[] args)  
     {  
       long freeBytesForUser;  
       long totalBytes;  
       long freeBytes;  
       Console.WriteLine("Free space in this directory:");  
       if (Win32.GetDiskFreeSpaceEx(@"C:\", out freeBytesForUser, out totalBytes, out freeBytes))  
       {  
         Console.WriteLine("Free user bytes: " + freeBytesForUser.ToString());  
         Console.WriteLine("Free total bytes: " + totalBytes.ToString());  
         Console.WriteLine("Free bytes: " + freeBytes.ToString());  
       }  
       Console.ReadLine();  
     }  
   }  
 }  

Reference: https://stackoverflow.com/questions/137255/how-can-i-determine-if-a-remote-drive-has-enough-space-to-write-a-file-using-c

Header Text Resizing on Scroll

This is a useful tool for any modern website UI with content that you want to showcase (and get the header content out of the way as much as you can).

CodePen: https://codepen.io/radagast27/pen/JqWENL

Source: This feature works by having JavaScript detect when a certain scroll position from the top of the web document (50 in this case) has been reached at which point a resize animation of the header text and area takes place. Then, the reverse happens when that same scroll position is reached while scrolling back up.

 <!DOCTYPE html>  
 <html>  
 <head>  
 <meta name="viewport" content="width=device-width, initial-scale=1">  
 <style>  
 body {   
  margin: 0;  
  font-family: Arial, Helvetica, sans-serif;  
  color: white;  
  background-color: black;  
 }  
 #header {  
  background-color: green;  
  padding: 30px 30px;  
  color: lightblue;  
  text-align: left;  
  font-size: 74px;   
  font-weight: bold;  
  position: fixed;  
  top: 0;  
  margin-bottom:5%;  
  width: 100%;  
  transition: 0.4s;  
 }  
 </style>  
 </head>  
 <body>  
 <div id="header">Hello.</div>  
 <div style="margin-top:190px; padding:15px 15px 250px 15px;">  
   <p>Lorem ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.Lorem ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.Lorem ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur   
      tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.Lorem ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.Lorem ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.Lorem ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.Lorem ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.Lorem ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.Lorem ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.Lorem ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.Lorem ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.Lorem ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. ipsum dolor dummy text sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.  
  </p>  
  <br />  
  The end.  
 </div>  
 <script>  
 window.onscroll = function() {scrollFunction()};  
 function scrollFunction() {  
  var hdr = document.getElementById("header")  
  if (document.body.scrollTop > 50 || document.documentElement.scrollTop > 50) {  
   hdr.style.fontSize = "30px";  
   hdr.style.height = "2%";  
  } else {  
   hdr.style.fontSize = "74px";  
   hdr.style.height = "10%";  
  }  
 }  
 </script>  
 </body>  
 </html>  

Reference: https://www.w3schools.com/howto/tryit.asp?filename=tryhow_js_shrink_header_scroll

Common Exploits and How They Work

Man in the Middle: "an attack where the attacker secretly relays and possibly alters the communications between two parties who believe they are directly communicating with each other."-Wkipedia.

A MITM or MTM attack does not happen at the source or destination but rather along the route between them. To prevent this type of attack one must ensure the integrity of their network. This includes enabling only the latest recommended security protocols, ensuring SSL cannot be impersonated, and general configuration of network firewalls and routing equipment to ensure no unauthorized user can ever connect to your router or any other interception point.


SQL Injection: "An SQL injection is a computer attack in which malicious code is embedded in a poorly-designed application and then passed to the backend database. The malicious data then produces database query results or actions that should never have been executed." -Techopedia.com

Cross-site Scripting (XSS): "XSS enables attackers to inject client-side scripts into web pages viewed by other users." -Wikipedia

For a basic example, if UserA is logged into some secure that is authenticating each UserA HttpRequest via a key that UserX can obtain- (perhaps by successfully MITM'ing)- UserX can then impersonate User A by using UserA's authentication key to craft HttpRequests containing malicious scripts that run automatically on User A's browser.



Buffer Overflow Attack:

Buffer Overflow attacks cause a memory buffer boundary (stack or heap) to be exceeded and memory pointers to be overwritten to point to attackers own malicious functions instead of the normal user, machine or OS instructions.

Intrusion detection systems can be used to mitigate this type of attack by alerting Network Administrators if any irregular/bad actor is on the network. Some well-known buffer overflow attacks: https://www.cypressdatadefense.com/education-training/buffer-overflow-attacks-need-know/


Rootkit Deployment: "A rootkit is a clandestine computer program designed to provide continued privileged access to a computer while actively hiding its presence." -Veracode.com

The unique feature of rootkits is their design to be undetectable. In Windows systems for instance, rootkits will override Win32 API methods that the OS uses to verify the integrity and authorization of certain method calls, etc. In this way, rootkits can allow a malicious program to run in the background, undetected by most normal system checks. Sysinternals' RootKitRevealer can show you if your machine is affected by any rootkit-based malware.

Sony Music infamously used rootkits: https://blogs.technet.microsoft.com/markrussinovich/2005/10/31/sony-rootkits-and-digital-rights-management-gone-too-far/

The following are some of the more well-known Exploits:

StuxNet: Uncovered in 2010, this severely malicious virus spread through Windows USB "Autorun" feature and mostly affected an Iranian nuclear enrichment plant. StuxNet was a self-propagating worm hidden via root level masking. The machine operators had no forewarning that StuxNet was installed and configured to ruin several centrifuges. StuxNet was developed jointly by US and Israeli cyber actors. Shockingly, the Iran centrifuge destruction was "just a test" of StuxNet's power. It successfully stalled Iran's nuclear enrichment program for up to several years according to NY Times researchers: https://www.nytimes.com/2011/01/16/world/middleeast/16stuxnet.html


WannaCry: "The WannaCry ransomware attack was a May 2017 worldwide cyberattack by the WannaCry ransomware cryptoworm, which targeted computers running the Microsoft Windows operating system by encrypting data and demanding ransom payments in the Bitcoin cryptocurrency." - Wikipedia 

WannaCry affected a number of targets (and successfully extorted a number of ransoms) including Britain's National Health Institute.


Sony Music: Sony's privates network was compromised by a group calling themselves "Guardians of Peace". The attack apparently happened simply from one of the attackers obtaining admin credentials through email phishing. Embarrassing internal email correspondence and future film material was leaked to the public. Ultimately, North Korean Park Jing Hyok has been charged for the Sony attack as well as the WannaCry ransom attack: https://www.pbs.org/newshour/nation/north-korean-programmer-charged-in-sony-hack-wannacry-attack


Spectre: In 2018 this hardware-based attack method altered the way microprocessors perform a  basic branch speculation function which leads to side effects that include revelations of what was in the process instruction, private data, etc. Although this attack is not remotely exploitable, it is likely to go undetected unless systems are maintained and secured diligently.



Cybersecurity Organizations and Resources:

OWASP Free and Open Security Community

Offensive Security

Microsoft Security

Android Security

Linux Security

Useful Wireshark Filters, Tips

Wireshark is a network traffic (packet) analyzer that is used for troubleshooting network issues and debugging applications at the network layer. With Wireshark you can usually isolate the device or communication link responsible for abnormal behavior in a network-based application.

Wireshark's UI can tell you a plethora of information about packets and their relationships; the key is isolate the WS info you need to solve your specific problem

The following are some of Wireshark's more useful tools for developers looking to get close inspection on network communications and data/packet movement:

Search for strings in packets: frame contains "local"

Display Filter reference: https://wiki.wireshark.org/DisplayFilters

Display Filter with Regex: frame matches "[B@]\w+" && frame.len < 55

Analyze existing pcaps: https://www.netresec.com/?page=PcapFiles

Follow Streams: Right-click packet and select "Follow" to isolate conversations.

Filler for only your IP: Filter only your local IP for src or dest to isolate your machine's traffic if you are on a switch.

Bad TCP setup: https://www.davidsudjiman.info/2018/02/08/capturing-bad-tcp-packets/

Regular Expressions

Regular expressions are a strings of (regex) code and matching (character) references used for finding patterns of characters within some defined contiguous string of characters (ie. forms validation, an IDE identifying opening and closing tags in markup code or a terminating semicolon in Java/C#). A more official definition describes regex as:

"In computing, regular expressions provide a concise and flexible means for identifying strings of text of interest, such as particular characters, words, or patterns of characters. Regular expressions (abbreviated as regex or regexp, with plural forms regexes, regexps, or regexen) are written in a formal language that can be interpreted by a regular expression processor, a program that either serves as a parser generator or examines text and identifies parts that match the provided specification."

 This diagram breaks down the gist of how an expression works- see the reference below for full list of matching modifiers and syntax

Having more than a passing familiarity with regex can be a tremendous help when doing any kind of development related to the identification of patterns in character/symbol data. When you understand how each component works you can very easily begin to build your own expressions to suit your particular application's pattern matching needs.

Below are some common regular expressions:

Alphanumeric Strings and Chars
^\w+$

Username (16 characters)
/^[a-z0-9_-]{3,16}$/

Hexadecimal ID
/^#?([a-f0-9]{6}|[a-f0-9]{3})$/

Email
/^([a-z0-9_\.-]+)@([\da-z\.-]+)\.([a-z\.]{2,6})$/

URL
/^(https?:\/\/)?([\da-z\.-]+)\.([a-z\.]{2,6})([\/\w \.-]*)*\/?$/

ipV4 Address
/^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$/

HTML Tag
/^<([a-z]+)([^<]+)*(?:>(.*)<\/\1>|\s+\/>)$/

XML Tags
Don't use Regex, use something like XPath


Just remember that the brackets are regex matching patterns, the curly braces are index positioners, the parens surround regex "capture groups" and the other syntax denotes flags for how much to match and other special conditions to be applied to the match/regex expression. Go here to practice and learn more: https://regexr.com/

The above illustrates Regex in 2 Capture Groups, One for alpha chars and the other for numeric chars

Reference: https://code.tutsplus.com/tutorials/8-regular-expressions-you-should-know--net-6149

Neo4j Graphs: Creating and Querying Edges and Nodes

First it is helpful to understand the basic premise behind Graph Theory, which is the foundation of these "Pairwise relations between objects" that we can store and explore in graph databases. The mathematical representation is the formula:

A graph (G) is equal to the connections of its entity nodes aka "vertices" (V) with its relationship edges (E)

Swiss mathematician Leonhard Eueler was looking for a solution to a puzzle on the relationships between land masses and bridges in the Prussian city of Konigsberg. This was the genesis of Graph Theory:

"The geometry of position, now known as Graph theory"

Graph database technology is useful for analyzing complex relationships and relationship behaviors in myriad scenarios not limited to:

• Computer Networking
• Spread of Gossip
• Fraud Detection
• Forensic Investigations
• Flight Mapping/GPS
• Population Growth
• Spread of Infectious Disease
• Hierarchy Visualization

It is also vital component for the analysis of (increasingly) unstructured data. By unstructured we typically mean data that is not easily modeled in a traditional relational hierarchy but may still have common properties and so still have relationship value. It is estimated that around 80-90% of an organization's data is unstructured.

Example of a couple simple graph relationships

Relationships in graph data are formed by the edge tables which signify a relationship between two different entities or "nodes" which are stored in node tables. Instead of RDBMS FK/PK and other check constraints, the relationships are defined in the edge tables; all of the properties that one would want to query via CQL to find things are stored in JSON metadata inside the node tables instead of the columns-for-each-property that defines relational data architecture.

Node Tables: Represent a data entity

Edge Tables: Store relationships between nodes

For the RDBMS purists and skeptics out there, I recommend this quote about Node, Graph and other alternative data processing paradigms vs. the traditional RDBMS OLTP and OLAP models:

"The NOSQL acronym is: Not Only S Q L. NOSQL solutions are not a replacement or successor for RDMBS systems, nor were they ever intended to be. They are useful tools to be used for specific purposes. They are to be used as part of an organisation’s data management solution and not as a total replacement for the existing solution". -Simon Munro

It is important to note that graph data can be queried in much the same way as SQL through a graph-specific query language called CQL (cipher query language) as you will see in the following demonstration.

For a quick demonstration on how easy it is to get up and running with this technology we will be using Neo4j which you can download here: https://neo4j.com/download/


Walkthrough of Neo4j:
Fire up the Neo4j Desktop client, click "New" and then click the "Add Graph" button to create your first graph database. Once the db has been created, click the play icon to start it up (it must be running for Neo4j browser to connect).

This is the Neo4j main home screen from which you can create and connect to graph databases and projects

Then click the "Neo4j Browser" button to launch the graph data browser for creation and visual exploration of graph data relationships.

Next, in the Neo4j browser, click "Jump into Code" and simply follow the prompts to begin creation and querying of graph data.

Once, in the Neo4j tutorial, you can simply follow the prompted instruction widgets.

I would continue with instructions but the rest is refreshingly self-explanatory. You first create a movie database that contains actors, directors and movies stored in Node (entity) tables and the types of relationships between these Nodes stored in Edge (relationship type) tables.

One thing to note is that to execute a CQL command in the browser, you must hold down CTRL+Enter. Alternatively you can click the play button to execute CQL.

As you continue with the tutorial you will wind up with something like this when you get to the step that has you execute CQL to find all Node entities that are within 4 degrees (or node "hops") of the actor Kevin Bacon:

4 degrees of separation from Kevin Bacon...

Once you get comfortable with CQL syntax it is relatively easy to start modeling and creating your own graph database structures which can help you and/or your company to analyze some of the unstructured and semi-structured data that is hard to extract value from with traditional RDBMS.

Bigtime kudos to the Neo4j team on making this so straightforward and simple to learn and get up
and running with a new technology so fast. I've never seen a technology tutorial like it.

As you can see, there is tremendous potential value in exploring data relationships that don't necessarily fit neatly into traditional RDBMS/hierarchical databases but are no less useful a tool to have in an organization's data analysis arsenal.


References:

https://www.mssqltips.com/sqlservertip/5007/sql-server-2017-graph-database-query-examples/

https://www.youtube.com/watch?v=gXgEDyodOJU

https://www.red-gate.com/simple-talk/sql/t-sql-programming/experiments-with-neo4j-using-a-graph-database-as-a-sql-server-metadata-hub/

https://www.youtube.com/watch?v=mVWn8k49mAQ

Securitization

Securitization is the creation and sale of pieces of debt from a pool of similar debt assets. It is a way for banks to take a group of home mortgage loans for instance, and cut the asset group into pieces or "tranches" that can be sold as MBSs (mortgage backed securities) on the open market.

Lots of touch points in this interesting "value abstraction" process

While many investment banks who used this financial implement in the run-up to the Great Recession have been strongly criticized for not vetting assets thoroughly enough in the origination process, the process of securitization will always be a method for asset holders to convert an illiquid asset like a group of home mortgages or consumer credit card debt into something (or rather "some things") that can be more easily packaged, bought and sold on the open market.


Reference: https://blog.bankex.org/paving-the-way-from-securitization-to-tokenization-ac0187ba6d48

Options, Calls and Puts

In finance, an option is a contract which gives the buyer the right, but not the obligation, to buy or sell an underlying asset at a specified price prior to or on a specified date, known as the "expiry date". An option contract typically requires an upfront payment for the option, called the premium.

A call option, also referred to as a "call" in finance jargon, gives the buyer the right to buy the underlying asset at an agreed-upon price on a specific date or within a specified period of time.

A put option, also referred to as a "put", gives the buyer the right to sell the underlying asset at an agreed-upon price on a specific date or within a specified period of time.

Calls give the right to buy, puts give the right to sell

The important characteristic of options contracts is that they give the right- not the obligation- to buy or sell an asset at some agreed upon price on or before the option's contract expiration date. The option holder can simply walk away from the option to buy or sell if she or he decides it is no longer in their best interest.

Options are another asset class, and when used correctly, they offer many advantages that trading stocks and ETFs alone cannot (namely the ability to decide not to exercise the option if the value of the underlying asset being bought or sold changes significantly (in the wrong direction) for the option holder before the expiry date, for instance).

Options are different from futures contracts in that option contracts give the right to buy or sell on or before some date, while futures contracts represent an obligation to buy or sell on some date.

With options, financial traders can lock in future gains if an asset value is expected to (and does) rise in value above their call price, and conversely can stem future losses if an asset value is expected to (and does) drop in value below their put price.


References:

https://investinganswers.com/financial-dictionary/optionsderivatives/option-2049

https://www.fool.com/investing/options/options-the-basics.aspx

The Infamous Story of ENRON

The story of Enron is a story of greed and how a Houston-based energy company rocketed to the top echelon of Corporate America before losing everything.

From stodgy Oil & Gas merger, to high-flying corporate giant, to an astonishing demise

Formed from the merger of Houston Natural Gas and InterNorth in 1985, Enron began with humble roots. Kenneth Lay was an enterprising economics graduate from Missouri who learned the ropes of the oil and gas business early while obtaining his PhD in economics in 1970 and working his way up to management at InterNorth before it was purchased by HNG.

For years the company had solid (if not spectacular) results and even overcame a couple near-fatal financial disasters that resulted from oil futures and origination guarantees deals gone bad. An almost overly-proud Harvard MBA from Illinois, Jeff Skilling joined Enron's ranks after several years of consulting for the energy giant as part of Enron's cozy relationship with McKinsey and Company.

Enron's fatal flaw was the belief that accounting "creativity" can permanently hide fraud 

In time, Skilling became COO and began to call for the mass hiring of elite MBA types and math gurus which he transformed into his "complex deal making" army. He became particularly close with Enron's oddball finance and accounting veteran Andrew Fastow who paired the brains of Jeff's army with the creativity of accounting fraud to make Enron appear, at least to investors and banks, as an extravagant capital-generating machine.

Fastow and his crack team of corporate fraudsters developed a network of shell companies known as SPEs or "special purpose entities" and used these as vehicles for hiding losses and booking fictitious deals- to the tune of several billion dollars of imaginary capital and unreported losses. Quarter after quarter, when Enron divisions were struggling to "hit the numbers" that Wall Street analysts expected- Andy would step in to save the day with his SPE magic that- at least temporarily- made bad news go away.

Another favorite method of Fastow and Skilling was to use "mark to market" accounting treatment of their energy deals. Meaning that they reported- as current income- all estimated future income of the life of the deal- for virtually all the deals they did. This is great when things are going good but it is an obviously untenable situation. While Enron was flashing the gaudy mark to market income figures to the Street, the future required them to actually service those deals- and never book another accounting profit as the entire deal's income has already been reported.

Enron's pursuit of Wall Street's favor made a mockery of their Code of Ethics

Enron, which had once been a company with deep roots in Oil & Gas and was hands-on in developing pipelines and sourcing fossil fuels for delivery contracts, was now in the business of trading on energy futures that bore little to no resemblance to true tangible "present values". Everything was speculation. Everything was reduced to hedges and bets. Nothing was real anymore. And it all collapsed under the weight of its own obfuscation.

Sure there were other reasons Enron collapsed. There was the comical Enron Broadband Services which tried to take on the early internet giants like AOL, and went..... nowhere. There were notorious global deals in places like India and England that became financial albatrosses which only Fastow's shell games could attempt to mask- for a time. But it was really just simple greed and criminal accounting.

Jeff Skilling Harvard MBA abstract mastermind, avoider of details and implementation

Even the once-proud accounting firm Arthur Anderson would be brought down by the fall of Enron and eventually file for bankruptcy. They had some protestations early on about the use of SPEs and the anachronistic manner of applying profits and losses, but ultimately they went along with and signed off on the grossly improper financial reporting.

The Justice Department, the SEC and FBI had long been looking at the company by the time Enron's offices were raided on January 22nd, 2002. What followed was the trial and conviction of several Enron executives including Fastow, Skilling and Lay who were sentenced for an assortment of fraud and conspiracy charges related to the heart of the scandal.

Andy Fastow was given a reduced 6 year sentence after agreeing to cooperate and testify against his former bosses. He was released from prison in 2011 and is now a popular speaker at business ethics and accounting fraud conferences.

Skilling received 24 years in federal prison for his role. He was released to a Texas half-way house on August 30th of 2018.

Ken Lay died of a heart attack while awaiting sentencing.

The biggest losers of Enron's demise were Enron employees and common stockholders who bet big on Enron's future

The timeline, web of deceit and cast of characters in this tragedy is truly fascinating. Rebecca Mark, Ken Rice, Lou Pai, and so many more interesting personalities are woven into this spectacular story that is told best by the people who (literally) wrote the book. For a comprehensive look into this business debacle, the award-winning book and documentary can be found here:

The Smartest Guys in the Room book by Bethany McLean and Peter Elkind

ENRON: The Smartest Guys in the Room

In the end, this was a tragedy of obscene hubris and ultimate humility. The ironic thing is that they had a solid business model and were it not for the lies that enabled inflated financial reporting, Enron- albeit a smaller and less glamorous Enron- would likely still be in business today.