Showing posts with label software development. Show all posts
Showing posts with label software development. Show all posts

You need to eat your own dog food

I recently realized how useless my custom .NET method, "Sonrai.ExtRS.ReferenceDataService.GetGoogleNews(string searchTerm)", really was.

I had referenced Sonrai.ExtRS via Nuget and began trying to use it to improve the display in scrolling news links, in another application I maintain, tickertapes.net.   

It was useless! I mean a complete waste of code. It returned potentially useful data related to the searchTerm parameter- data from the GoogleNews API. But that is about it. It literally returned the entire XML response. 🤦‍♂️


This was the raw oil, but not the refined gasoline a consumer of a Nuget package expects...


I was parsing all this XML in the tickertapes.net app (why I do not know or care to remember), and so all the work necessary to wrangle the XML response to produce the needed news link collection was on the consumer of the Sonrai.ExtRS Nuget package- not good!

What the Nuget library should do, is all the work. What it should return, to be actually useful, is something structured like a collection of strings, each one representing a single news article, with the news headline being the text for the news article link.

And so I moved back to Sonrai.ExtRS to correct this unfortunate oversight.  

  public static async Task<List<string>> GetGoogleNewsWithLinks(string search)  
  {  
    HttpClient client = new HttpClient();  
    var content = await client.GetStringAsync(string.Format("https://news.google.com/rss/search?q={0}", search));  
    var parser = new HtmlParser();  
    var document = parser.ParseDocument(content);  
    var newsItems = document.All.Where(m => m.LocalName == "title").ToList();  
    var linkItems = document.All.Where(x => x.LocalName == "link").ToList();  
    var newsLinkItems = new List<string>();  
    for (int i = 0; i < newsItems.Count; i++)  
    {  
      newsLinkItems.Add("<a href='" + linkItems[i].NextSibling!.NodeValue + "' target='_blank'>" + newsItems[i].InnerHtml + "</a>");  
    }  
    return newsLinkItems;  
  }  


And this provides the Nuget client with something it can actually use, "turnkey"/out-of-the-box.

A list of HTML links which we can actually use; this is more like it.


The moral of the story is one that is as old as business and manufacturing: you must eat your own dog food. If there are problems or areas that need attention it is best that you find this information out before your product is released into the wild and a customer discovers the bug (or in this case, the uselessness), thus sullying your reputation as a business and software provider.

Test, test, test- always unit and/or integration test every user interaction and data movement for every story/path imaginable or support-able.

But there is nothing that replaces simply using your own product the way it is used by real users. And really using it for the purpose it was made. What you discover may help you shore up previously unknown problems and/or inspire you to make something useful that you would never otherwise think of, unless you were thinking from a user's perspective.


Reference: https://www.nuget.org/packages/Sonrai.ExtRS


PS: Imagine if James Newton-King never used NewtonSoft.Json for his own de/serializations? Or if Stack Overflow never used Dapper for the SO app/site? You need to use your creations right at the ground level (as a user/client) to verify that the functionality you designed in the abstract exactly matches how things will play out in concrete reality. Thoughts.. 💭

How to use .NET User Secrets in MSTest classes

When developing unit and integration tests, we don't necessarily want to share the secret keys and values we use as credentials for APIs, databases, etc.

So similarly to how we implement User Secret functionality in Program.cs, we can implement User Secrets and set test class variables that use the secrets via the MSTest classes' constructor (ReferenceDataTests() below):

  public static string upsId = "";  
  public static string upsSecret = "";  
  private IConfiguration _configuration { get; }  
  
  public ReferenceDataTests()  
  {  
    // set your API ids and secrets in UserSecrets (right-click project: "Manage User Secrets")  
    var builder = new ConfigurationBuilder()  
      .AddUserSecrets<ReferenceDataTests>();  
    _configuration = builder.Build(); 
    
    var secretVals = _configuration.GetChildren().ToList();  
    upsId = secretVals.Where(x => x.Key == "upsId").First().Value!;  
    upsSecret = secretVals.Where(x => x.Key == "upsSecret").First().Value!;   
  }  




An SSRS IFrame/CORS infinite redirect loop error and a quick and easy solution

The redirect loop looks like this and, in Edge, will display the error message: "[domain] redirected you too many times"

If you are trying to render the SSRS ReportViewer control within an <iframe>, you may run into a CORS issue that manifests in a series of 302 (Found) responses and an infinite redirect loop between ReportViewer control (ReportViewer.aspx) and Logon.aspx.

As of SSRS 2022, without an explicit instruction to allow CORS, ReportViewer cannot be rendered within an <iframe> on an origin different than the origin of the report server.

If you are using custom authentication, the solution is easy enough. Just add cookieSameSite="None" and enableCrossAppRedirects="true" to the authentication <forms> tag in the report server's web.config.

 <authentication mode="Forms">  
   <forms loginUrl="logon.aspx" name="sqlAuthCookie" cookieSameSite="None" timeout="60" path="/" enableCrossAppRedirects="true" requireSSL="true">  
   </forms>   
 </authentication>  

You may also need to enable CORS in your client app. In ASP.NET Core 8, this can be achieved through the following in your application startup code:
 app.UseCors(builder => builder  
 .WithOrigins("https://localhost", "https://[domain]")  
 .AllowAnyMethod()  
 .AllowAnyHeader());  

Dichotomies

Apropos of nothing I thought about some interesting dichotomies and wanted to share my perspective. It is interesting how often there are effectively 2 sides to a coin.


Religious Faith/Religious Dogma - Some folks live the "spirit" of the Word and resist structure vs. others, who follow strict adherence to "the letter" of the Word.

Subjective/Objective - You have here opinion and feeling (the supernatural) vs. facts and laws of nature.

Romantic/Classical - similar to the Subjective/Objective dichotomy (see also, "Zen and the Art of Motorcycle Maintenance").

Liberal/Conservative - The ideology of inclusion and change vs. the ideology of exclusion and stasis.

Conglomerate/Individual Co. - Whereas many corporations sought to obtain economies of scale through M&A in 80s, 90s and 00s, General Electric proved- (sold GE Capital, spun off GE Healthcare, GE Aviation and GE Power into 3 individual new companies)- that the conglomerate model doesn't always stand the test of time.

Thin client/Fat client - It makes no sense to be all one or all the other (an overburdened SPA app or an inflexible server-side only app). But we keep moving between one (thin client terminals in the 70s and 80s) and the other (fat client Home PCs in the 90s and 00s). And we've moved back to thin client again with Azure, AWS and GPC and the omnipresence of SaaS. But at the same time, fat-client SPA apps are as popular as ever... I guess we have fat client UIs and thin client APIs.

Imperative/Declarative code - Instructions that read like a book vs. instructions that read like a mathematical proof.

Monolithic app/Microservices - a very heavy Swiss-Army knife vs. a bunch of lightweight kitchen knives.

Software/Hardware - Recipes vs. the raw food itself.

Socialism/Capitalism - The idea that everyone is the same and should be reduced to (or propped up by) exactly such vs. the idea that everyone is incompatibly unique and should be uniquely catered to and provided maximum personal freedoms that can- like assault weapons and ammunition- come at the expense of the greater community.

Urban/Rural - Those from the densely populated cities and suburbs vs. those from the sparsely populated country towns.

Introvert/Extrovert - He or she who expends a great deal of energy socializing and finds solace in silence vs. he or she who is energized by interpersonal social connection and is not comfortable alone for extended periods of time.

Centralization/Decentralization- Putting the heart of a system at the center with dependent, often necessarily-generic/homogenized nodes vs. putting the heart on the nodes themselves, at the expense of (losing) ease of simultaneous node (state) synchronization and a "single source of truth/golden records".



Code Readability vs. Code Golf

All developers aim to achieve expressiveness in their code. Which is to say we strive to ensure our code is readable and that we make efficient use of our code so that things are not repeated and the behaviors of interacting objects or functions are clear and easy to understand even for a new developer who has never previously seen the code base.

But sometimes, "expressiveness" can come at a cost. Having readable code means different things to different development teams. But a clear sign of unreadable code (no matter how much IDE screen real estate is saved) is something like this oddly nested structure of if statements.



This code is pretty objectively horrible no matter how you slice it..



If no other developers can understand your code, it won't matter how elegant your code is; it will become difficult to find people who can quickly read it, understand it, and change it. Lacking such ability to transfer the codebase knowledge can turn a great piece of software into millstone for an organization once all of the original developers have left the building.

So in comes my (*biased) opinion of code golf and how it impedes code readability. It's neat and a fun way to enhance your programming skills and knowledge of a language or framework; but it serves as a terrible template for producing readable code that your colleagues can fairly easily look at and understand, or at least debug and determine how the abstractions interact and what pieces of functionality are contained in (or spread across) which files.

Code Golf: https://code-golf.iohttps://codegolf.stackexchange.com  (have fun, you can learn lots)


ASM, clear for some developers; not most:
             global    _start
             section   .text
_start:   mov       rax, 1                    ; system call for write
             mov       rdi, 1                    ; file handle 1 is stdout
             mov       rsi, message         ; address of string to output
             mov       rdx, 13                 ; number of bytes
             syscall                               ; invoke operating system to do the write
             mov       rax, 60                 ; system call for exit
             xor       rdi, rdi                ; exit code 0
             syscall                           ; invoke operating system to exit

             section   .data
message:  db        "Hello, World", 10      ; note the newline at the end

-vs-

Java, unclear 1-liner:
public class StringStatsArray { private final String[] stats; public StringStatsArray(String[] a) { stats =
  a; } public String toString() { String ret = "{"; for (String check: stats) { ret += "\"" + check + "\", "
  ; } ret = ret.substring(0, ret.length() - 2) + "}"; return ret; } public double averageLength() { double
  sum = 0; for (String check: stats) { sum += check.length(); } return sum / (double) stats.length; } public
  int search(String target) { for (int i = 0; i < stats.length; i++) { if (stats[i].equals(target)) { return
  i; } } return -1; } public int sortStatus() { if (stats.length <= 1) { return 1; } if ((int) stats[0]
  .charAt(0) < (int) stats[1].charAt(0)) { for (int i = 0; i < stats.length - 1; i++) { if ((int) stats[i]
  .charAt(0) > (int) stats[i + 1].charAt(0)) { return 0; } } return 1; } else if ((int) stats[0].charAt(0) >
  (int) stats[1].charAt(0)) { for (int i = 0; i < stats.length - 1; i++) { if ((int) stats[i].charAt(0) <
  (int) stats[i + 1].charAt(0)) { return 0; } } return -1; } else { for (int i = 0; i < stats.length - 1; i
  ++) { if ((int) stats[i].charAt(0) < (int) stats[i + 1].charAt(0)) { for (int j = i; j < stats.length - 1;
  j++) { if ((int) stats[j].charAt(0) > (int) stats[j + 1].charAt(0)) { return 0; } } return 1; } if ((int)
  stats[i].charAt(0) > (int) stats[i + 1].charAt(0)) { for (int j = i; j < stats.length - 1; j++) { if ((int
  ) stats[j].charAt(0) < (int) stats[j + 1].charAt(0)) { return 0; } } return -1; } } return 1; } } }

-vs-

C# Minecraft client, clear code:
using (var world = JavaWorld.Connect())
{
    world.PostToChat("Hello from C# and .NET Core!");
    var originBlock = world.GetBlockType(0, 0, 0);
    world.PostToChat($"Origin block is {originBlock}.");
}



"Code formatting is about communication, and communication is the professional developer’s first order of business." -Robert C. Martin (Uncle Bob)


Even complex languages can be simplified somewhat by using logical naming conventions to identify what the entities are and what operations are doing.

Clear and consistent naming conventions of all members within the code of an application go a long way for making your code open to collaboration and/or extension by others. In much the same way people prefer different kinds of poetry, we developers prefer certain programming languages or styles (.NET vs. Java vs. NodeJS vs Python vs. C and C++; Message Queues vs. RDBMS; OO vs. FP) more or less than others. We should however, as far as possible, attempt to make our code clearly formatted and named for anyone to easily interpret. We should be as clear and consistent as possible and only comment when explaining the behavior in variable or function names alone won't suffice.

And this is all not to be confused with just good old fashioned best practices encompassed in the excellent book Clean Code by Robert Martin. Write clean code. Write understandable code with well-named members etc. And for the love of software development (esp. open-source) having a fighting chance to reach its potential in this early part of the 21st century, please do not write code in such a way that serves only to prove how clever you are or obsfuscate what is really going on for reasons unknown. "Security through obsfucation" is a real thing, but it's not a good way to manage a team code base.



Assume that the developers who inherit your code are going to need a little guidance



*true story, I was once interviewed first on the phone, and then on-site for an interesting .NET/SQL Server Dev opening with a major league baseball team. I did relatively well in the 3+ hour interview which included live coding exercises and meeting with all of my potential future colleagues. I went home that day pretty confident that I'd get the job, which in my experience is usually a sure sign that I won't get the job.

The last step for me before the hiring decision was made, arrived in the form of an email which let me know that the other final candidate had submitted a better solution than mine to the final and most difficult coding challenge but only by a couple bytes (16 or 24 bits or something) and that I was going to be given a chance to resubmit my code if I could make it smaller.

The final challenge was to demonstrate some extreme DRY by writing a program that would output the lyrics to "Old McDonald had a farm" in the smallest possible source code file. So, logically I abracted away all the individual characters and repeating character sequences and did my best to make the program as small as possible.

I declined to attempt any improvements. I honestly didn't know how to make it smaller, but more to the heart of the matter, this code golf barrier for entry made me wonder if there were other development practices within the IT department of this organization which I would not be a fit for; so it was probably mutually beneficial that I dropped the ball.

Due to a few .exe filesize bytes... I came in 2nd place and lost out on my dream contract/job. All of this because of my failure to hit a hole-in-one in that darn code golfing test.

My (C#) solution looked somewhat similiar to this:

void x(string c, string d){var a="Old MacDonald had a farm";var b=", E-I-E-I-O";var f=" a ";var g=" there";Debug.WriteLine(a+b+",");Debug.WriteLine("And on that farm he had"+f+c+b+",");Debug.WriteLine("With"+f+d+" "+d+" here and"+f+d+" "+d+g+",");Debug.WriteLine("Here"+f+d+","+g+f+d+", everywhere"+f+d+" "+d+",");Debug.WriteLine(a+b+"!");


I think the actual (C?) solution with the smallest filesize was something like this: 

o(char* x, char* y)
{
    char* f=
        "Old MacDonald had a farm, E-I-E-I-O,\n"
        "And on that farm he had a %s%.13s"
        "With a %s %s here and a %s %s there,\n"
        "Here a %s, there a %s, everywhere a %s %s,\n"
        "%.35s!\n";

    printf(f,x,f+24,y,y,y,y,y,y,y,y,f);
}

Ah well, we live and learn. Development is not a series of 1-upping and out-dueling one another with battles of cleverness. It's an ongoing communication, elaboration and collaboration experiment wherein the ultimate goal is for everyone to understand the domain problems and solutions, domain modeling for the problem abstractions and a standard domain language which removes ambiguity and enables "1000ft" knowledge for even the most tertiery of interested stakeholders.

Some areas of code (for instance the data layers or the UI or the file I/O, etc.) will invaribly be understood by some team members more or less than others. And a team architect will understand everything, though not at the low, "weeds" level of the dev who is implementing a particuliar story or piece of functionality.

Code like you might die tomorrow and someone will have to pick up everything from scratch. Enable frictionless configuration for setting up the local IDE and environment for the first time. Create a useful, concise README.md that includes everything needed to get up and debugging and an explanation of anything that is completely or somewhat non-standard. Include unit and e2e integration tests that 2x-check runtime interactions and clearly demonstrate all behaviors and interactions. 

Above all else: make your application instructions so readable and elegant that perusing the source code is not unlike reading a book and immidiately understanding and visualizing each entity and operation and what typical usage and operation looks like. It's really hard, but it is important for the future that we all get it right and continuously improve our code's understandability.

Happy developing, all.







"All the quirks and buggy behaviours we can exploit in a code-golf challenge allow us to identify with less effort code that will otherwise behave unexpectedly or cause trouble."
...
I’d argue that code-golf can be greatly beneficial as long as we keep our mind in the game and not try using it where it doesn’t belong." -Juan Cortés

Very, very true points in favor of code golfing for fun and deep exploration (even if not the default for real world implementations).


A Timeless Book (by Jerry Fitzpatrick)

TL; DR;

This book gets right to the point of the common aspects required for building great software and is too valuable for any software development professional not to read; and it contains just 172 pages! 😃


Read this book.


Plan Before Implementing

Proper and properly named abstractions to match problem domain and purpose. Thoroughly documented and well-understood behavioral interactions among all accessible components and at minimum a solid high level understanding of all dependencies; know when not to re-invent the wheel and instead utilize a proven 3rd party or open source tool.


Keep it Small

YAGNI ("You ain't gonna need it") is a very real thing. Development teams should focus on working prototypes that can be ironed out for production vs. forever "ideal" implementations that never make it to an actual user's screen.


Write Clearly

Treat a rough draft as a rough draft by encouraging code reviews and frequent minor refactoring to achieve code clarity for the next developers who will inevitably read (and need to understand in order to change or extend) your logic in the future.


Prevent Bugs

Meticulously control scope and member access and understand all ramifications of behavioral code points within the application. Every area of variation/mutation/state change should be covered by tests that are both named and implemented to clearly demonstrate the function/class/behavior being tested.

All tests are important for confidence/assurance in ongoing development but TDD (writing tests before the implementation a la Kent Beck) is not a replacement for good, cohesive design. That being said, I think the TDD backwards approach can help start great design discussion among developers when the design is somewhat unclear (tests, particularly broken tests will expose the innards that most warrant discussing).


Make the Program Robust

Utilize agreed-upon code formatting and pattern implementation standards and non-antagonistic code reviews to ensure these standards are being followed.

Utilize information hiding and useful categorization and encapsulation that extends but does not lend itself to potential client issues.

Do not try/catch everything and log everything unless necessary for compliance or a particuliar initiative being monitored. Make Exceptions very visible (but exit the program gracefully if the exception is not "handleable") and integrate tests to prevent the same (preventable) issues from recurring. Instead use try/finally and let exceptions bubble up to the surface application code.

Exceptions that are allowed to be handled and logged hide problems in the code that need immediate attention.

Use solid CI/CD and automated testing that tests for easily definable application behaviors.


Prevent Excess Coupling

Favor atomic initialization: Initialize everything all at once versus incremental composition.

Discourage unnecessary extensibility points and instead expose what needs to be exposed with as little information sharing as possible.

Strive for immutable members wherever possible.

Cohesive, self-evident abstractions are of utmost importance.


SUMMARY

The concepts so succinctly covered in Timeless Laws of Software Development apply to all types of code (imperative or declarative) and all languages.

While I've read many great software books and all have helped me become a better developer, no text has struck me as so plainly obvious and concise at explaining good software design concepts in such an easy-to-grok manner.

For a worthwhile read, check out Timeless Laws of Software Development by Jerry Fitzpatrick

.NET 5+: the Future of .NET

Microsoft doesn't want to restrict the growth of their highly popular .NET development framework by requiring that it only run on Windows OS and that its components rely on Windows-based components like ASP.NET which relies on IIS.

Enter .NET Core 1, 1.1, 2, 5 (6)+: the Future of .NET.


One .NET Framework to rule them all? No. But a new, unified, cross-platform .NET Runtime? Yes.


With .NET 5 (and coming in hot down the pike, .NET 6), developers can use the languages (C#, F#, VB and C++/CLI) and framework (.NET) they are familiar with, to build applications that can run on a Windows, Linux or Mac OS. While .NET Core 1 and 2 lacked several expected features and felt fairly unsupportable, .NET 5 feels more familiar and addresses most issues (this is just the networking improvements made).

Formerly known as .NET Core 1, .NET Core 1.1 and then .NET Core 2, the future for the .NET Framework and associated runtime libraries will fall under the umbrella of what is simply called ".NET" (5 (current release), 6 (preview), with intermediate minor versions one would expect) as of the time of this writing. ASP.NET Core will continue to be labeled with the "Core" designation for now with the .NET version tagging the runtime (ASP.NET Core Runtime 5, ASP.NET Core Runtime 6, etc.).

There do exist more-than-slightly-subtle differences. For instance ASP.NET Core Runtime 5 projects are structured differently and Global.asax.cs has been replaced with Program.cs and the accompanying Startup.cs which contain the Main entry point and provide spaces for application configuration methods that execute only when your program is initially run (this includes functions to respond to certain events like a unhandled/last chance exception handling, etc.).

    public class Startup
    {
        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public IConfiguration Configuration { get; }

        // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)
        {
            services.AddDbContext<ApplicationDbContext>(options => options.UseSqlServer(Configuration.GetConnectionString("DefaultConnection")));
            services.AddDefaultIdentity<IdentityUser>(options => options.SignIn.RequireConfirmedAccount = true).AddEntityFrameworkStores<ApplicationDbContext>();
            services.AddControllersWithViews();
            services.AddRazorPages();
            services.AddMvc().AddViewLocalization(LanguageViewLocationExpanderFormat.Suffix);
            services.AddLocalization(options => options.ResourcesPath = "Resources");
            services.Configure<RequestLocalizationOptions>(options =>
            {
                options.SetDefaultCulture("en-US");
                options.AddSupportedCultures("en-US", "es-CO", "fr-FR");........

An example of .NET 5 ASP.NET Startup.cs which replaces most Global.asax.cs functionality; Program.cs handles all else

 

In sum total, .NET 5 is basically a framework with the same functionalities that the .NET Framework had, with new namespaces and methods and a few slightly different design paradigms for ASP.NET and other project templates that use .NET 5 code. But it compiles to binary code that can run crossplatform- to a Linux distro or OSX27 or regular old Windows 10, 11, 12, etc.).

.NET 5 seems to favor better encapsulation/information hiding (less extensibility but fewer things to keep track of that can go wrong!) but ASP.NET Core still tends to have the magic string architecture that is its double-edged sword. Everything is a balance; never can be 100% fast, cheap and customizable.

Expect lots more to come now that .NET has it's versioning, cross-platform integration and road map in order



Virtually everything is customizable if you want to use your own implementation of a SqlClient RDBMS provider or Authentication provider. Setting up social media authentication is relatively simple. For relational database access, Dapper is a nice balance between ADO.NET with pure SQL statements and an ORM for object population/deserialization, etc. and compiles to .NET Standard which can be referenced and compiled by a .NET 5 project.

With SQL Server moving to Linux world I imagine many other useful services that used to rely solely on Windows-OS will follow: native MS Office, MS Dynamics and related dependencies, IIS, etc.

Where .NET really shines is in its ability to compile/integrate .NET code libraries (with the notable exception of server-side WCF)  built with .NET Framework 4.5-4.8 as well as Xamarin, Mono, Unity and other previous .NET-related runtime libraries. It uses what is called .NET Standard (a collection of commonly used and .NET 5-compatible .NET Framework  members) to achieve this bridging between .NET and all .NET Framework versions preceding it.

Perhaps not all of that old .NET 4.5-4.8 legacy code will need to go to waste.


All SDKs under one roof; what is not to like? Almost as great as "Write Once.. Run Anywhere" 


Long-term support for .NET 5's predecessor, .NET Framework will continue for versions 4.6.2 and above for the foreseeable future while prior versions have either already ended LTS or will be ending LTS soon, per Microsoft:

Support for .NET Framework 4, 4.5, and 4.5.1 ended on January 12, 2016.

Support for .NET Framework 4.5.2, 4.6, and 4.6.1 will end on April 26, 2022. Customers and developers must have completed the in-place update to .NET Framework 4.6.2 or later by April 26, 2022 to continue receiving technical support and security updates.

"Next Big" Software Religiosity and The Go-nowhere Rush

There is far too much religious extremism in information technology these days. And there have always been camps (extreme anti-Microsoft sentiment or its sad corporate counterpart: disdain, fear and suspicion of all things open-source)- but these days it has gotten to the point where sensible, cheap, reliable, proven solutions that everyone on the team understands- are thrown out in favor of chasing the next big thing that some bigshot at some big conference declared was going to be the next, next, next "big thing".


This image does have its merits..


Amid all the continuous rush to be cutting edge despite understanding what that edge can do for you and having a strong data foundation to build upon with that new cutting edge thing- it doesn't matter what tools are out. You are still stuck with ideas and not programs.

Design and develop with what works for your particular team and project and within the context of the environments of your stakeholders (if all but 2% of your customers use Android then the iPhone version of your app may not be as important as you think). Above all else, make sure you understand the domain knowledge behind the data your application will be persisting and passing around. That (the data understanding) is the heart of every program that stores, processes, transmits or even simply reads/prints/paints- any kind of communication.

Data sense-making and software development is hard work. And it's not done in a void. I suggest reading Stephen Few's "Big Data, Big Dupe" which is a little paperback containing 90 some pages of important wisdom for this modern rapid-fire information age that pre-empts knowledge of data in favor of slogans and metrics about data.

In short, the essence of this book is that if you have say 10TB of crap data that is always causing ETL failures that your personnel spend countless hours trying to correct... you may indeed have "big data" per some misguided tech journalist's definition... but you still have crap data-- understand your data before you try understanding how best to fit it inside of the newest shiny box.

Take also for example message queues and their usage in modern web application development. There seems to be a lot of misunderstanding about what MQ is and even some who claim this is a new technology (MSMQ has been around since Windows '95; IBM MQ has been in use since 1993). Basic email has operated on a publisher/subscriber (ICMP or SMTP) messaging queue paradigm that works in much the same way as modern MQ implementations (minus some bells and whistles)- since the early 70's.

These things aren't as complicated as they seem but they are complicated. And it's perilous to keep jumping from new trick to new trick whilst ignoring foundational, timeless software principles.

I would go so far as to say it is injurious to current and future generations of software developers to keep focusing on buzzwords, zooming out and away from the hard-but-necessary work of understanding the data, and then wondering why the tool or framework flavor of the year did not save the day.

ChartJS for Data Vizualiizations

I came across ChartJS about 2 years ago while debugging code from another, similar data visualization technology inside an AngularJS app.


ChartJS is a flexible JavaScript data visualization framework which allows for some pretty powerful integrations and customizations


The concept is you have a "<canvas>" DOM element, which you transform into a ChartJS chart via some JavaScript initialization. After that your tasks are simply finding the data you want to render and deciding the options of exactly how you want the chart visual to appear.

You can do some really neat and dynamic stuff with ChartJS.

I have used a lot of charting frameworks, and it does not get more flexible or simple than this:
 <html>  
 <head>  
 <script src="https://cdn.jsdelivr.net/npm/chart.js@2.8.0"></script>  
 </head>  
 <body>  
 <div>  
 <canvas id="myChart" style='background-color:darkgray; width:100%; height:100%;'></canvas>  
 </div>  
 <script>  
 var ctx = document.getElementById('myChart').getContext('2d');;  
 var chart = new Chart(ctx, {  
   type: 'line',  
   data: {  
     labels: ['16_Qtr1', '16_Qtr2', '16_Qtr3', '16_Qtr4', '17_Qtr1', '17_Qtr2', '17_Qtr3', '17_Qtr4', '18_Qtr1', '18_Qtr2', '18_Qtr3', '18_Qtr4', '19_Qtr1', '19_Qtr2', '19_Qtr3', '19_Qtr4', '20_Qtr1', '20_Qtr2', '20_tr3', '20_Qtr4', '21_Qtr1', '21_Qtr2', '21_Qtr3', '21_Qtr4','22_Qtr1', '22_Qtr2', '22_Qtr3', '22_Qtr4', '23_Qtr1', '23_Qtr2', '23_tr3', '23_Qtr4'],  
     datasets: [{  
       label: 'Some random quartley demo data..',  
       backgroundColor: 'black',  
       borderColor: 'lime',  
       data: [40.2, 72.88, 47.1, 22, 54.43, 52.18, 17.1, 52, 67.2, 54.88, 64.1, 78, 67.2, 55.88, 58.1, 57, 50.2, 52.88, 57.1, 62, 74.43, 62.18, 67.1, 72, 77.2, 74.88, 74.1, 78, 77.2, 75.88, 78.1, 77, 70.2, 72.88, 77.1, 62, 64.43, 62.18, 67.1, 72, 67.2, 54.88, 44.1, 28, 27.2, 25.88, 38.1, 37, 40.2, 42.88, 44.1, 52, 54.43, 52.18, 67.1, 82, 87.2, 84.88, 84.1, 88, 87.2, 95.88, 108.1, 127]  
     }]  
   },  
    "options": {  
       "legend": {"position": "bottom"}      
     }  
 });  
 </script>  
 </body>  
 </html>  


Reference: https://www.chartjs.org/

Over-engineering in Software

TL; DR; - build the app; not your imagined future state of the app

Implementing what you know you will absolutely need before all else is the best course of action for development


This is a great brief guide on how it happens and how to prevent or at least diminish the phenomena that is borne out of a desire to do things exact and right- but ignores that fact that some things never are evaluated and time is.... finite. (in (virtually) every project, we only have so much time to get the bells to ring and the whistles to whistle...

That interface to allow for a potential future Python SDK, putting a small proprietary API on Swagger with loads of documentation and test calls just "because" might not be a road you want to go down yet..

Creating a .NET Standard portable class library of the core functionality with a full suite of tests because you think it may be Nugetized in the future... is not something you should do until the app is already engineered and built well, out the door and humming along swimmingly!

Over-engineering is essentially development of designs and/or writing code that solves problems that do not currently exist.


BugZilla learned the hard way that "Future-Proofing" taking too literally can become over-engineering and waste lots of time for stuff never to be used/needed



Bugzilla has dealt with it and a former developer left a very concise and telling quote from their experience:
Some people think design means, “Write down exactly how you’re going to implement an entire project, from here until 2050.” That won’t work, brother. That design is too rigid–it doesn’t allow your requirements to change. And believe me, your requirements are going to change. 
Planning is good. I should probably do more planning myself, when it comes to writing code. 
But even if you don’t write out detailed plans, you’ll be fine as long as your changes are always small and easily extendable for that unknown future. 
-Max


TAKEAWAY:

  • "Business requirements never converge. They always diverge."
  • "At the beginning, it’s best to stick to the KISS principle over DRY."
  • Favor less abstraction for things that are currently 100% exactly known

Long ago one of my software engineering colleagues at a large Milwaukee-based industrial company (thanks John Ignasiak!) taught me a way to use what is known as "soft coding" in relational SQL to use data abstraction tables in order to abstract data types until run-time and multi-purpose data types and relations whilst keeping all values in the same table.


Just never enough generalizations and assumptions to make in software development... lol


I am not a big fan of multi-purposing things. But sometimes it is the only way and in this case of a legacy system with an old reporting db, it was the only way for time we had available.

Use the tools for the task they were made for.

Use the data type (VARCHAR for instance) for the data it was made for, not for dynamic SQL and hard-coded but dynamically inferred data types. The latter usages are almost always symptomatic of a terrible, terrible database design or something that your design just cannot accommodate without major restructuring of relational hierarchy and dependencies.

That all being said, SQL soft-coding (and dynamic SQL) is awesome and has some really powerful use cases where it is the perfect approach for the task at hand.

Build the app by the requirements as they currently exist, not by your imagination of how the future may or may not affect the app.

Two principals that have helped many a developer over the years are the acronyms:


  • KISS (Keep It Simple)
  • YAGNI (You Ain’t Gonna Need It)



Reference: https://www.codesimplicity.com/post/designing-too-far-into-the-future/

Neat UI Effects with jQueryUI and Animate.CSS

So you want neat animation effects on your UI? Seems like it might be a tricky implementation, right?

Animate.css makes it easy to implement modern UI animation effects that make for better user experience/cues


It could not be simpler with jQuery UI and other open source options. Here is an easy sample to pick up and run with from jQuery.com:

jQuery UI Effects
 <!doctype html>  
 <html lang="en">  
 <head>  
   <meta charset="utf-8">  
   <meta name="viewport" content="width=device-width, initial-scale=1">  
   <title>jQuery UI Effects - Effect demo</title>  
   <link rel="stylesheet" href="//code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css">  
   <link rel="stylesheet" href="/resources/demos/style.css">  
   <style>  
     .toggler {  
       width: 500px;  
       height: 200px;  
       position: relative;  
     }  
     #button {  
       padding: .5em 1em;  
       text-decoration: none;  
     }  
     #effect {  
       width: 240px;  
       height: 170px;  
       padding: 0.4em;  
       position: relative;  
     }  
       #effect h3 {  
         margin: 0;  
         padding: 0.4em;  
         text-align: center;  
       }  
     .ui-effects-transfer {  
       border: 2px dotted gray;  
     }  
   </style>  
   <script src="https://code.jquery.com/jquery-1.12.4.js"></script>  
   <script src="https://code.jquery.com/ui/1.12.1/jquery-ui.js"></script>  
   <script>  
     $(function () {  
       // run the currently selected effect  
       function runEffect() {  
         // get effect type from  
         var selectedEffect = $("#effectTypes").val();  
         // Most effect types need no options passed by default  
         var options = {};  
         // some effects have required parameters  
         if (selectedEffect === "scale") {  
           options = { percent: 50 };  
         } else if (selectedEffect === "transfer") {  
           options = { to: "#button", className: "ui-effects-transfer" };  
         } else if (selectedEffect === "size") {  
           options = { to: { width: 200, height: 60 } };  
         }  
         // Run the effect  
         $("#effect").effect(selectedEffect, options, 500, callback);  
       };  
       // Callback function to bring a hidden box back  
       function callback() {  
         setTimeout(function () {  
           $("#effect").removeAttr("style").hide().fadeIn();  
         }, 1000);  
       };  
       // Set effect from select menu value  
       $("#button").on("click", function () {  
         runEffect();  
         return false;  
       });  
     });  
   </script>  
 </head>  
 <body>  
   <div class="toggler" style="margin-top:200px;">  
     <div id="effect" class="ui-widget-content ui-corner-all">  
       <h3 class="ui-widget-header ui-corner-all">Effect</h3>  
       <p>  
         Etiam libero neque, luctus a, eleifend nec, semper at, lorem. Sed pede. Nulla lorem metus, adipiscing ut, luctus sed, hendrerit vitae, mi.  
       </p>  
     </div>  
   </div>  
   <select name="effects" id="effectTypes">  
     <option value="blind">Blind</option>  
     <option value="bounce">Bounce</option>  
     <option value="clip">Clip</option>  
     <option value="drop">Drop</option>  
     <option value="explode">Explode</option>  
     <option value="fade">Fade</option>  
     <option value="fold">Fold</option>  
     <option value="highlight">Highlight</option>  
     <option value="puff">Puff</option>  
     <option value="pulsate">Pulsate</option>  
     <option value="scale">Scale</option>  
     <option value="shake">Shake</option>  
     <option value="size">Size</option>  
     <option value="slide">Slide</option>  
     <option value="transfer">Transfer</option>  
   </select>  
   <button id="button" class="ui-state-default ui-corner-all">Run Effect</button>  
 </body>  
 </html>  


And one I whipped up from Animate.css...:
 <html>  
 <head>  
  <script src="https://code.jquery.com/jquery-3.4.1.min.js" integrity="sha256-CSXorXvZcTkaix6Yvo6HppcZGetbYMGWSFlBw8HfCJo=" crossorigin="anonymous"></script>  
  <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/animate.css/3.7.2/animate.min.css">  
  <script>                 
      $( document ).ready(function() {  
           myDiv = document.querySelector('#myDiv')       
           $("#buttonBounce").on("click", function () {  
                removeAllClasses();  
                myDiv.classList.add('animated', 'bounce')  
           });  
           $("#buttonSwing").on("click", function () {  
                removeAllClasses();  
                myDiv.classList.add('animated', 'swing')  
           });            
           $("#buttonWobble").on("click", function () {  
                removeAllClasses();  
                myDiv.classList.add('animated', 'wobble')            
           });            
           $("#buttonTada").on("click", function () {  
                removeAllClasses();  
                myDiv.classList.add('animated', 'tada')  
           });       
           $("#buttonBounceOut").on("click", function () {  
                removeAllClasses();  
                myDiv.classList.add('animated', 'bounceOutLeft')  
           });            
           function removeAllClasses(){  
                myDiv.classList.remove('animated', 'bounce', 'wobble', 'bounceOutLeft', 'tada', 'swing');  
           }  
      });       
  </script>  
 </head>  
 <body style="background:#000000;">  
 <div id="myDiv" style="margin-top:100px; margin-left:300px; width:250px; background:aqua; font-weight:bold; font-family:Arial; border: 2px solid lime; padding: 10px; border-radius: 25px;">~Something to animate~</h1>  
 <br/><br/>  
 &nbsp;&nbsp;<input type="button" id="buttonBounce" class="ui-state-default ui-corner-all" value="Bounce" />  
 </br><br/>  
 &nbsp;&nbsp;<input type="button" id="buttonTada" class="ui-state-default ui-corner-all" value="Tada" />  
 </br><br/>  
 &nbsp;&nbsp;<input type="button" id="buttonSwing" class="ui-state-default ui-corner-all" value="Swing" />  
 </br><br/>  
 &nbsp;&nbsp;<input type="button" id="buttonWobble" class="ui-state-default ui-corner-all" value="Wobble" />  
 </br><br/>  
 &nbsp;&nbsp;<input type="button" id="buttonBounceOut" class="ui-state-default ui-corner-all" value="Bounce Out" />  
 </br><br/>  
 </body>  
 </html>  


This is what the Animate.css code above will look like..


Happy coding!


Codepens: 

https://codepen.io/radagast27/pen/pooavXd

https://codepen.io/radagast27/pen/xxxYboM



References:

https://jqueryui.com/effect

https://github.com/daneden/animate.css

ASCII Art with Python

Have you ever wanted to convert an icon or photo to ASCII art? Fortunately there are lots of tools to do this out there. I looked around for one that works with Python 3+ and found a succinct, modular script that can output ASCII art from image files like the following:


The script is from Paul Bourke. There is very clear commenting on each step. Essentially, an image loaded from the "--file" argument is divided into a grid and each position of the grid is assigned certain ASCII character whose gray scale level most closely matches the average for that area of the image. I simply modified line 146 and 150 to demonstrate on console vs. the outfile.txt default.

Save the following as ascii.py:
 #Python code to convert an image to ASCII image.  
 import sys, random, argparse  
 import numpy as np  
 import math  
 from PIL import Image  
 # gray scale level values from:  
 # http://paulbourke.net/dataformats/asciiart/  
 # 70 levels of gray  
 gscale1 = "$@B%8&WM#*oahkbdpqwmZO0QLCJUYXzcvunxrjft/\|()1{}[]?-_+~<>i!lI;:,\"^`'. "   
 # 10 levels of gray  
 gscale2 = '@%#*+=-:. '   
 def getAverageL(image):  
      # get image as numpy array  
      im = np.array(image)  
      # get shape  
      w,h = im.shape  
      # get average  
      return np.average(im.reshape(w*h))  
 def covertImageToAscii(fileName, cols, scale, moreLevels):  
      # declare globals  
      global gscale1, gscale2  
      # open image and convert to grayscale  
      image = Image.open(fileName).convert('L')  
      # store dimensions  
      W, H = image.size[0], image.size[1]  
      print("input image dims: %d x %d" % (W, H))  
      # compute width of tile  
      w = W/cols  
      # compute tile height based on aspect ratio and scale  
      h = w/scale  
      # compute number of rows  
      rows = int(H/h)  
      print("cols: %d, rows: %d" % (cols, rows))  
      print("tile dims: %d x %d" % (w, h))  
      # check if image size is too small  
      if cols > W or rows > H:  
           print("Image too small for specified cols!")  
           exit(0)  
      # ascii image is a list of character strings  
      aimg = []  
      # generate list of dimensions  
      for j in range(rows):  
           y1 = int(j*h)  
           y2 = int((j+1)*h)  
           # correct last tile  
           if j == rows-1:  
                y2 = H  
           # append an empty string  
           aimg.append("")  
           for i in range(cols):  
                # crop image to tile  
                x1 = int(i*w)  
                x2 = int((i+1)*w)  
                # correct last tile  
                if i == cols-1:  
                     x2 = W  
                # crop image to extract tile  
                img = image.crop((x1, y1, x2, y2))  
                # get average luminance  
                avg = int(getAverageL(img))  
                # look up ascii char  
                if moreLevels:  
                     gsval = gscale1[int((avg*69)/255)]  
                else:  
                     gsval = gscale2[int((avg*9)/255)]  
                # append ascii char to string  
                aimg[j] += gsval  
      # return txt image  
      return aimg  
 # main() function  
 def main():  
      # create parser  
      descStr = "This program converts an image into ASCII art."   
      parser = argparse.ArgumentParser(description=descStr)  
      # add expected arguments  
      parser.add_argument('--file', dest='imgFile', required=True)  
      parser.add_argument('--scale', dest='scale', required=False)  
      parser.add_argument('--out', dest='outFile', required=False)  
      parser.add_argument('--cols', dest='cols', required=False)  
      parser.add_argument('--morelevels',dest='moreLevels',action='store_true')  
      # parse args  
      args = parser.parse_args()  
      imgFile = args.imgFile  
      # set output file  
      outFile = 'out.txt'   
      if args.outFile:  
           outFile = args.outFile  
      # set scale default as 0.43 which suits  
      # a Courier font  
      scale = 0.43   
      if args.scale:  
           scale = float(args.scale)  
      # set cols  
      cols = 80   
      if args.cols:  
           cols = int(args.cols)  
      print('generating ASCII art...')  
      # convert image to ascii txt  
      aimg = covertImageToAscii(imgFile, cols, scale, args.moreLevels)  
      # open file  
      f = open(outFile, 'w')  
      # write to <console or> file ..f.write(row + '\n')  
      for row in aimg:  
           print(row + '\n')   
      # cleanup  
      f.close()  
      print("ASCII art written to console...") # %s" % outFile)  
 # call main  
 if __name__ == '__main__':  
      main()  


Using Windows CMD, navigate to the directory where your ascii.py script is located. Then execute the following:

ascii.py --file C:\somedir\logo.jpg --cols 150


Referencehttp://paulbourke.net/dataformats/asciiart/

DevOps as Culture

DevOps is an emerging IT role but also a culture; and changing culture does not happen overnight

  • Elevate empowerment and give courage to people to "raise hand" and volunteer to fix things. Even if they are wrong- interest in areas outside one's primary role is good for the entire DevOps culture.
  • Shared accountability - no blame game between developers, PMs, testers, QA, etc.
  • Do not want to have to be dependent on "lone genius" or "firefighter"- need to share and transfer knowledge.
  • Offer time to learn. Encourage hacking for new features and for hardening security and find bugs.
  • Embrace failures in retrospectives to prevent repeated mistakes on future work.
  • Provide the right incentives to motivatve the values you want to reward: reward delivery of quality vs. fire fighting.
  • Understand "value streams" (esp. value stream bottleneck and how can we optimize all constraints) to know where to spend time accordingly.
  • Focus on CONSTRAINTS.
  • Avoid Configuration Drift- Config changes should cascade to all environments (QA, DEV, TEST, STAGE, PROD).
  • Automate the Path to Production.
  • Use pull-based systems so that people integrate each others changes and learn how everything works in unison/concert.
  • People should not fear for their jobs- Systems Admin becomes more important not less so, in DevOps
  • DevOps is how you work not just what you buy or what tech you are using.
  • Total adoption happens in stages/iterations.
  • Traditional Project and Documentation mindset is outmoded, outdated, and disconnected from a living IT mission.
  • Lack of DevOps leads to waste and waiting (waiting for ppl with the right skills to work on things vs. having team that can easily shift contexts or frameworks/languages).
  • SHARE KNOWLEDGE AND EMPOWER COLLEAGUES TO DO MULTIPLE TASKS AND UNDERSTAND SOME OR ALL ASPECTS OF MULTIPLE RESPONSIBILITIES OF THE SOFTWARE PROCESS CHAIN.


NLog for Flexible Application Logging in .NET

As much as I have enjoyed working with log4net in the psat NLog works much more seamlessly for .NET apps (it has good, solid abstractions)

From Nuget Package Manager, you can find and reference NLog and the accompanying NLog.Config which simplifies setup. To configure NLog logging, simply point the required properties (you need at least a logFile variable, a target and a rule to get started) to your desired values in the config file that NLog.Config creates in your project root  (NLog.config):

Configuration:

 <?xml version="1.0" encoding="utf-8" ?>  
 <nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"  
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  
    xsi:schemaLocation="http://www.nlog-project.org/schemas/NLog.xsd NLog.xsd"  
    autoReload="true"  
    throwExceptions="false"  
    internalLogLevel="Off" internalLogFile="c:\temp\nlog-internal.log">  
  <variable name="logFilePath" value="C:\NLog\IDG.log" />  
  <targets>  
   <target name="logfile"  
     xsi:type="File"  
     fileName="${logFilePath}"  
     layout="${longdate}  LEVEL=${level:upperCase=true}: ${message}"  
     keepFileOpen="true" />  
  </targets>  
  <rules>  
   <logger name="*" minlevel="Info" writeTo="logfile" />  
   <logger name="*" minlevel="Warn" writeTo="logFile"/>  
  </rules>  
 </nlog>  
This example uses a log file target in C:\NLog directory; you can utilize a wide variety of logging targets to broadcast app errors

Source Code:

 using Microsoft.VisualStudio.TestTools.UnitTesting;  
 using NLog;  
 namespace ExtRSTests  
 {  
   [TestClass]  
   public class NLogTests  
   {  
     private static Logger logger = LogManager.GetCurrentClassLogger();  
     [TestMethod]  
     public void TestLoggerToFile()  
     {  
       logger.Warn("Something in the app happened that may indicate trouble ahead....");  
       logger.Error("Uh-oh. Something broke.");  
     }  
   }  
 }  
With the logging levels, log formatting (timestamps) and abundant integration options, NLog is a complete logging solution

With NLogger you can implement logging for an array of targets including file, email, database and 3rd party integrations (ie. send message to Slack channel if logger generates any "Error" or "Fatal" level log message).


Reference: https://www.infoworld.com/article/3144535/how-to-work-with-nlog-in-net.html