I'm a software developer and consultant living in Cary, North Carolina.

Extended scancodes in sendinput


A while ago I wrote a virtual KVM program that would allow you to drive the keyboard and mouse of other computers (even cross-platform), using a single host machine. Communication was done over the network. I kept track a the "virtual" position of the mouse, and if you veered into the screen position of another computer I would eat all keystrokes / mouse movement via a hook and replay it on the other machine.
<https://github.com/michaelwda/OutsideTheBox/blob/master/OTB.Core/Hook/Platform/Windows/WindowsGlobalHook.cs>

Pretty fun code to write, for a toy project. I ran into some interesting things regarding keyboard input. Really the best method here is to go as low-level as possible. On Windows I decided to use SendInput, but it would probably be even better to write a virtual keyboard/mouse driver.

When using SendInput, you can specify that it's a keyboard input and send the scancodes directly. This gives you lots of power to replay keystrokes exactly, but it was a little tricky when trying to replay extended scancodes.

From some old comments of mine:

```C#
/*
* The extended-key flag indicates whether the keystroke message originated from one of the additional keys on the enhanced keyboard.
* The extended keys consist of the ALT and CTRL keys on the right-hand side of the keyboard; the INS, DEL, HOME, END, PAGE UP, PAGE DOWN, and arrow keys in the clusters
* to the left of the numeric keypad; the NUM LOCK key; the BREAK (CTRL+PAUSE) key; the PRINT SCRN key; and the divide (/) and ENTER keys in the numeric keypad. The extended-key flag is set if the key is an extended key.
*/
```

If you're trying to replay an extended scancode via sendinput, you have to pass an array of scancodes with the first one being 0Xe0.

```C#
 public override void SendKeyDown(Key key)
        {
            int tscancode;
            VirtualKeys tvk;
            int tflags;
            var keyup = false;
            var altDown = KeyboardState.IsKeyDown(Key.AltLeft) || KeyboardState.IsKeyDown(Key.AltRight);
            bool extended;
            WinKeyMap.ReverseTranslateKey(key, keyup, altDown, out tscancode, out tvk, out tflags, out extended);

            bool sysKey = (!altDown && key == Key.AltLeft) || (!altDown && key == Key.AltRight) || ((key != Key.AltLeft && key != Key.AltRight && altDown));

            var dwFlags = 0x0008;
            if (extended)
                dwFlags = dwFlags | 0x0001;

            var altdown = ((tflags) & ((int)KeyFlags.KF_ALTDOWN >> 8)) > 0;
            var dlgmode = ((tflags) & ((int)KeyFlags.KF_DLGMODE >> 8)) > 0;
            var menumode = ((tflags) & ((int)KeyFlags.KF_MENUMODE >> 8)) > 0;
            var repeat = ((tflags) & ((int)KeyFlags.KF_REPEAT >> 8)) > 0;
            var up = ((tflags) & ((int)KeyFlags.KF_UP >> 8)) > 0;

            KeyboardState.SetKeyState(key, true);

            NativeMethods.INPUT[] inputs;
            if (extended)
            {
                inputs = new[]
                {
                    new NativeMethods.INPUT
                    {
                        type = NativeMethods.INPUT_KEYBOARD,

                        u = new NativeMethods.InputUnion
                        {
                            ki = new NativeMethods.KEYBDINPUT()
                            {
                                wScan = (ushort) 0xe0,
                                wVk = (ushort) 0,
                                dwFlags = (ushort) 0,
                                dwExtraInfo = NativeMethods.GetMessageExtraInfo()
                            }
                        }
                    },
                    new NativeMethods.INPUT
                    {
                        type = NativeMethods.INPUT_KEYBOARD,

                        u = new NativeMethods.InputUnion
                        {
                            ki = new NativeMethods.KEYBDINPUT()
                            {
                                wScan = (ushort) tscancode,
                                wVk = (ushort) tvk,
                                dwFlags = (ushort) dwFlags,
                                dwExtraInfo = NativeMethods.GetMessageExtraInfo()
                            }
                        }
                    }
                };
            }
            else
            {
                inputs = new[]
                {
                    new NativeMethods.INPUT
                    {
                        type = NativeMethods.INPUT_KEYBOARD,

                        u = new NativeMethods.InputUnion
                        {
                            ki = new NativeMethods.KEYBDINPUT()
                            {
                                wScan = (ushort) tscancode,
                                wVk = (ushort) tvk,
                                dwFlags = (ushort) dwFlags,
                                dwExtraInfo = NativeMethods.GetMessageExtraInfo()
                            }
                        }
                    }
                };
            }

            NativeMethods.SendInput((uint)inputs.Length, inputs, Marshal.SizeOf(typeof(NativeMethods.INPUT)));
        }
```

I couldn't find this documented ANYWHERE except in a footnote of an image in some scanned documentation. Maybe it's in some manuals somewhere, but I had the hardest time figuring it out.

At some point I'm going to reboot and completely rewrite this project, especially now that C# has traits :)


How to run tika-server as a windows service using Apache Commons Daemon

 
I have submitted a PR to the Apache Tika project for this, we'll see if they merge it. <https://github.com/apache/tika/pull/324>

Using Apache Commons Daemon <https://commons.apache.org/proper/commons-daemon/> you can host tika-server as a windows service. However, tika-server has no static method for stopping the server, so once started it's impossible to stop the windows service. Not really ideal.

To fix this, you need build from source and add a stop method to TikaServerCli.java and to TikaServerWatchDog.java 

# TikaServerCli.java 
```java
static TikaServerWatchDog watchDog = null;

private static Options getStopOptions()
{
    Options options = new Options();
    options.addOption("preventSystemExit", false, "Prevent the stop method from calling system.exit, which will terminate the JVM. This is useful for integration tests.");
    return options;
}

private static void execute(String[] args) throws Exception {
        Options options = getOptions();

        CommandLineParser cliParser = new GnuParser();
        //need to strip out -J (child jvm opts) from this parse
        //they'll be processed correctly in args in the watch dog
        //and they won't be needed in legacy.
        CommandLine line = cliParser.parse(options, stripChildArgs(args));
        if (line.hasOption("spawnChild")) {
            /*TikaServerWatchDog watchDog = new TikaServerWatchDog();*/
            watchDog = new TikaServerWatchDog();
            watchDog.execute(args, configureServerTimeouts(line));
        } else {
            if (! line.hasOption("child")) {
                //make sure the user didn't misunderstand the options
                for (String childOnly : ONLY_IN_SPAWN_CHILD_MODE) {
                    if (line.hasOption(childOnly)) {
                        System.err.println("The option '" + childOnly +
                                "' can only be used with '-spawnChild'");
                        usage(options);
                    }
                }
            }
            executeLegacy(line, options);
        }
    }

public static void stop(String [] args) {
    // process service stop function
    if(watchDog != null)
    {
        watchDog.close();
    }
    try{
        Options options = getStopOptions();
        CommandLineParser cliParser = new GnuParser();
        CommandLine line = cliParser.parse(options, args);
        if (line.hasOption("preventSystemExit")) {
            return;
        }
    }       
    catch (org.apache.commons.cli.ParseException e){
        e.printStackTrace();
        LOG.error("Can't parse stop arguments: ", e);
        System.exit(-1);
    }  
    System.exit(0);
}
```

# TikaServerWatchDog.java 
```java
public void close()
{
    setChildStatus(CHILD_STATUS.SHUTTING_DOWN);
    LOG.debug("about to shutdown");
    if (childProcess != null) {
        LOG.info("about to shutdown process");
        childProcess.close();
    }
}
```

Once that's built, take your new jar file, and you'll be able to set it up with Apache Commons Daemon. 

- Download the windows specific binary <https://downloads.apache.org/commons/daemon/binaries/windows/>
- Extract. Rename "prunmgr.exe" to "tika-daemon.exe"
![install](/static/tika/1.png)

- I will usually place my jar file in the same directory
# Install as a windows service 
```
C:\source\tika\commons-daemon-1.2.2-bin-windows\amd64\prunsrv.exe //IS//tika-daemon ^
--DisplayName "Tika Daemon"  ^
--Description "Tika Daemon Windows Service" ^
--Classpath C:\source\tika\tika-server.jar ^
--StartClass "org.apache.tika.server.TikaServerCli" ^
--StopClass "org.apache.tika.server.TikaServerCli" ^
--StartMethod main ^
--StopMethod stop ^
--StartMode jvm  ^
--StopMode jvm ^
--StdOutput auto ^
--StdError auto ^
--Jvm "C:\Program Files\Java\jdk1.8.0_211\jre\bin\server\jvm.dll" ^
++StartParams -spawnChild 

```
Once installed, you can double click the renamed "tika-daemon.exe" and view the configuration. If the service fails to start, configure the logging options to inspect the output.
In my case I had trouble with both the JVM AND the archicture. Above you'll see I'm using the 64bit prunsrv.exe and I've manually set the JVM dll path.

`C:\source\tika\commons-daemon-1.2.2-bin-windows\amd64\prunsrv.exe`

`--Jvm "C:\Program Files\Java\jdk1.8.0_211\jre\bin\server\jvm.dll" ^`

![install](/static/tika/2.png)

Once done, tika will be up and running on the default port. Additional configuration can be provided inside the `++StartParams` options

![install](/static/tika/3.png)

AND... it works

![works](/static/tika/4.png)

On firing someone


I've been intending to write this for a while, but twitter didn't really seem like the correct platform and I had let my blog languish, so I just left it alone for a while.

I fired someone by in November. It was decisive, within our 90-day probationary period, and I'm certain it was the right call. I walked into the office that day knowing what I had to do, knowing it needed to be done, and it was still just terrible all-around. The thing about it is, the blame can be placed on both sides of the table, but honestly I feel like it's more on my side. Lots of failures at all levels.

- Poor hiring. I had a whole rubric that I neglected because I personally like the guy. But when it came right down to it, there was a significant skills mismatch. 
- Lack of structure and support to assist getting him up to speed. I was constantly out of the office and not around to help.
- It took him by surprise. Clearly just a complete breakdown in communication on my part. 

Lots to take away from that experience, but mostly for me was that clear communication and expectations are key. I'm still working on it.


2019 Recap


## 2018 Goal Recap
- _Less travel - the upcoming arrival of our baby girl has made this one really easy for me. I told all my clients, truthfully, that I don't want to travel very far when my wife is in here 3rd trimester. I am also not going to be travelling after she's here because my wife needs me home. My Raleigh client has me onsite 2 to 3 days a week in 2018, and the rest of the time will be spent remote. _
     * I'm happy to say that in 2019, I **significantly** reduced my travel. I was _in-town_ more, but that didn't neccesarily mean I was _home_ more. Even when I was home, a lot of the time that meant I was in my home office. 

- _Better time management - I cannot realistically continue to do late-night marathon code sessions. It's just not sustainable. I am planning to be very disciplined with my time-management in 2018. I'm also considering ways I can be held accountable when working from home on open-ended projects. Maybe I'll start live streaming myself in the office working? _
    * Surprisingly, I've actually gotten better at this, mostly because of constraints around having kids. It's just not feasible to work all night, and then be woken up at 6AM with a crying baby. I still had a few last-minute crunches, but overall it has been much better.

- _Less Coffee - I've gone a little overboard on the coffee and really need to cut back. In January I'm going to begin my journey._
    * After my trip to Korea, I actually kept this up for a while. I was all the way down to single cold-brew a day, but now I'm back up to old levels. It seems to be correlated with alcohol too. If I have a glass of wine at night, I really need that extra caffeine to get going. This turns into a cycle where I sleep restlessly because of the increased caffeine. 

- _Make 20% more, again. This will be interesting. I'm close to hitting a hard-cap based purely on the fact that there's only so many hours in the day. Even with the reduced travel, I will need to scale in other ways to accomplish this. I'm thinking productized services and possibly hiring another developer. _
    * I went the development / expansion route, a year late. 2018 was flat, 2019 saw a 35% bump. Hired a few guys, fired one. Still not sure if it all was worth it.

- _Work on the business rather than in the business. e-myth stuff._
    * Yep, but everything has tradeoffs

- _Spend time with my wife and new baby. The only way I can achieve this in the way I want is to have extreme discipline with my time-management. ​_
    * 2018 went really well, and then I got bored and hired a team in 2019. Still working on the transition to management.

- _Continue learning. This year I want to deep-dive on .NET Core, CSS grid, and React._
    * Only managed .NET Core, but this blog is React and flexbox, so that's something.

- _Launch something. I want to launch something public-facing of my own. I do it for clients all the time, I should do something for myself. _
    * Nope ☹

# 2019 / Early 2020 recap
I just migrated by blog to Next.JS and netlify and had to convert all my posts to markdown, re-reading them in the process. Reading my 2017 recap post was particularly interesting and resulted in a mix of feelings as I looked through my "goals". I'm proud of what I accomplished in 2018, 2019, and early 2020, but there's definitely some regrets. I think perhaps my biggest regret is that I jumped so hard into building a team and expanding, that I didn't stop to consider the tradeoffs. And there's always a tradeoff. 

1. Managing projects and the team
2. Expanding your individual technical skills
3. Being involved at home
4. Fitness and personal well-being

The above ordering is exactly how I prioritized in 2019. Managing the team, shipping code, lining up projects all took a front-row seat. And make no mistake, we did well. We shipped several huge projects last year, I really leveled up technically, and I made more money than ever. This all came with some serious downsides. Multiple all-nighters, I'm 10 lbs heavier than I'd like to be, I wasn't as available at home as I'd like. Tradeoffs. It might be because really I'm doing 2 or 3 seperate jobs: PM, Developer, Sales. 

May 2020, my son was born while we were stuck at home because of the pandemic. Interesting times. I'm so lucky to have been able to work from home and take a generous amount of time off, but it has been trying to say the least. Most notably, because of nature of my job, it's basically impossible to truly step away. Yes, I may be "away", but my phone is still buzzing. I may have built a team around me, but I'm still pivotal in all operations. That's probably my fault, if I'm honest. I did my best to empower others, but at the end of the day they didn't take the reigns and I ended up with a backlog of fires to put out every evening ... which gets exhausting. Combined with the fact that I have to be busy selling and keeping the pipeline full, just really tough. 

# 2020 Goals
- Reboot blogging, but focus on deeply technical topics. My few technical posts have always gotten the most traffic. I should double-down with a focus on things that aren't easily found online.
- Better sleep / less caffine. Take 2 on this, I may have to stop drinking completely to make it happen. I've found there's actually a downside though. The 6 months in 2019 that I stopped drinking, I found it absolutely impossible to turn off. Those 6 months were perhaps the most productive of the past 10 years, but I'm going to need a release mechanism. Maybe hiking or meditation. Something.
- Deeper dive into work that has more outsized leverage. What I mean is, I'm unhappy working on code that's being used by 100 people max. At one of my clients, my code is being run by 30k users. That's more like it. I'm really interested in the unexplored problem spaces (for me) around scaling that up to 300k or 3 million. 
- Get a new job to more quickly get the point above this one? 

I'm struck with the fact that the items I really want to focus on from my priorities above are 2, 3, and 4. That leaves item #1 (Managing projects and the team) completely on the table and I'm left wondering what to do about it. Covid + parental leave has also caused my entire backlog to evaporate, save for one big client with a yearly contract. All good things must come to an end, but I'm completely opposed to letting a good developer go because I'm ready to personally move on. It just seems so selfish. I'm not sure what to do, other than just complete honesty. Let him know my plans, the situation about the backlog and what I'm doing to line up additional work, and assist him with interviewing while keeping him gainfully employed in the interim. Meanwhile we'll just burn through the rest of the contract in Q3.

EF Core Soft-Delete


This post is a combination of two posts. I have removed the original, because my implementation of soft-delete had a bug that only occured once you started implementing more complex joins. I'd rather not be spreading broken code around.

This version works.
https://stackoverflow.com/questions/45096799/filter-all-queries-trying-to-achieve-soft-delete/45097532#45097532

I believe this was only so complicated because I wanted to be stubborn and apply the filter automatically by iterating across my modelBuilder. 

```C#
public static class EFFilterExtensions
{
    public static void SetSoftDeleteFilter(this ModelBuilder modelBuilder, Type entityType)
    {
        //this is using reflection to set the generic arguments for the SetSoftDeleteFilter
        //below. This let's us iterate across the entities in db-context and then call
        //by passing in the clrType
        SetSoftDeleteFilterMethod.MakeGenericMethod(entityType)
            .Invoke(null, new object[] { modelBuilder });
    }

    static readonly MethodInfo SetSoftDeleteFilterMethod = typeof(EFFilterExtensions)
        .GetMethods(BindingFlags.Public | BindingFlags.Static)
        .Single(t => t.IsGenericMethod && t.Name == "SetSoftDeleteFilter");

    public static void SetSoftDeleteFilter<TEntity>(this ModelBuilder modelBuilder)
        where TEntity : BaseModel
    {
        modelBuilder.Entity<TEntity>().HasQueryFilter(x => !x.Deleted);
    }
}
  
  protected override void OnModelCreating(ModelBuilder modelBuilder)
  {
      base.OnModelCreating(modelBuilder);
      foreach (var entityType in modelBuilder.Model.GetEntityTypes())
      {
          if (entityType.ClrType.IsSubclassOf(typeof(BaseModel)))
              modelBuilder.SetSoftDeleteFilter(entityType.ClrType);
      }
  }
  
  public abstract class BaseModel
  {
      public int Id { get; set; }

      public bool Deleted { get; set; }

      public DateTime DateCreated { get; set; } = DateTime.Now;

      public DateTime DateModified { get; set; } = DateTime.Now;
  }
  ```

Undersized Instances


I launched a rather large project on AWS using very small instances for the application servers and database (RDS). The application itself is seasonal, so the traffic spikes in December and then levels out. In December we had 250k unique page views.

It was kind of interesting using under-sized instances and seeing how things go wrong. On the application side of things it sailed along. MySQL is where all the problems seemed to be concentrated.

I spent a bit of time early in my career writing A LOT of SQL and within the past few years I've spent a bit of time leveling up my skills in regards to MS-Sql internals. Things like indices, waits, latches, how to use extended events, etc. So anyways, I just happened to have a specific set of skills to put out the MySql dumpster fire before it crashed the whole application. 

If anyone is interested, below is what deadlocks look like.

![](/static/undersized/deadlocks.jpg)

It makes me wonder what would have happened if I had over-sized these instances and been ignorant of sql internals. Would the application have just continued working? Maybe it would have kept trudging along, doing more heavy lifting than it should, while being just a little bit laggy. Maybe I would have thrown more resources at it. 

I can certainly empathize with someone who just wants to put their data somewhere and pull it back without having to worry about how a deadlock condition is going to exhaust their connection pool. I'm sure it's things like this that gave rise to BaaS platforms like firebase.

Successful Launch


It’s really rewarding to see a project go live. Late last year I build and launched an online portal for a client. Over the past 10 months we registered 14k users. These users are already customers of the client, but historically have had to mail in checks and paper forms.

Last week we launched the online equivalent of these checks and forms. I was the sole developer of the external-facing application. I was worried when we turned it live that we wouldn’t see any traffic for a while…but someone started using it literally within seconds. We’ve had thousands of people using it this week.

So, had a fun little trial-by-fire thing last week with : hitting bank CC authorization limits, DB indexing, deadlocks, deploys, etc. In general though, everything worked really well and it was a complete success. 5 more came through while I was typing this. We crossed $150k over the weekend!

![](/static/launch/Capture.png)

It’s a great feeling to work hard and have it pay off.

CLR Generics and Struct Boxing


The PR comments from @benaadams here are awesome. https://github.com/dotnet/corefxlab/pull/2358
This is likely basic knowledge needed for high-performance .NET, but it was new to me. Very interesting to see how defining a generic `<T>` can eliminate boxing/allocations.

The T parameter is explicitly being specified as an interface: `IBufferWriter<byte>;` so if you create that `BufferWriter<IBufferWriter<byte>>` and give it a struct as the T, then its boxed to the interface and stored in the `BufferWriter<IBufferWriter<byte>>` as the boxed `IBufferWriter<byte>` type.
So, if you explicitly specify the T, it will get boxed. If you specify it as a generic, and then call the generic constructor, it will not.

```C#
public static JsonWriter<TBufferWriter> Create<TBufferWriter>(TBufferWriter bufferWriter, bool isUtf8, bool prettyPrint = false) where TBufferWriter : IBufferWriter<byte>
{
     return new JsonWriter<TBufferWriter>(bufferWriter, isUtf8, prettyPrint);
}
```

ODWEK Middleware with Spring Boot

Recently a client was upgrading to CMOD V10.1 on the iSeries and we noticed that the CGI/Servlet support has been removed, after being deprecated in V9.0

*"OnDemand Web Enablement Kit (ODWEK) CGI/Servlet support no longer shipped: Support for CGI/Servlet was deprecated at V9.0 and has been removed from the product at V10.1."*

At this particular client, that I've wrote about before, we are doing some interesting things with pulling documents and text out of CMOD programatically. We have custom reports, email attachment functions, on the fly TIF conversion, etc. All of this is written in C# wrapped around the legacy CGI component.

Upgrading to V10.1 means we needed to re-implement the CGI functionality using the Java API. My first inclination was to use IKVM to convert the API to a C# dll and call it directly. I ended up hitting a brick wall with this related to some strange type marshalling and decided to implement the way IBM likely intended: with a java servlet.

I decided that if I was going to basically re-implement the CGI functionality, I would make it work the way I needed. Previously I was calling the CGI page and scrapping the HTML. Forget that now that I'm going to be in control, I'm going to return JSON!

# Install ODWEK
Our CMOD is running under the iSeries, but I decided to code the servlet for a tomcat instance running on windows. A coworker gave me a copy of ODWEK V9 for Windows. He didn't have a copy V10.1 yet, and I needed to get this done, so I went with it. I doubt the API surface will change too much between versions. He has also recently informed me that he upgraded to V10.1 on the iSeries, and my servlet environment running under Windows has continued functioning, so it doesn't appear like there's much of an issue interoperating.

Install the ODWEK for windows. Go on, I'll wait.

![install](/static/odwek/Install.png)

# Make ODWEK Work!

Don't get fancy yet. Write a plain old java program and make it work. Before you even start introducing the compexity of Tomcat, Spring, etc, let's just make a basic program work. This is absolutely neccesary because there's actually a fair amount of things you need to make work.

![three files](/static/odwek/3files.png)

I wrote a series of classes that will will query a folder. Compile each of these with javac. Notice how I reference my install path to the ODAPI jar file. You will need the java SDK installed for this.

I've uploaded each of these files to github for reference.

<https://github.com/michaelwda/ODWEKServlet/tree/master/PlainJava>


```bash
javac -cp "C:\Program Files\ibm\OnDemand Web Enablement Kit\V9.0\api\ODAPI.jar";. ODQueryResult.java

javac -cp "C:\Program Files\ibm\OnDemand Web Enablement Kit\V9.0\api\ODAPI.jar";. ODUtil.java

javac -cp "C:\Program Files\ibm\OnDemand Web Enablement Kit\V9.0\api\ODAPI.jar";. QueryFolder.java
```

The queryfolder is the program we care about. It's called with 3 parameters: FolderName Criteria Value

Now, try to run your program.
```bash
java -cp "C:\Program Files\ibm\OnDemand Web Enablement Kit\V9.0\api\ODAPI.jar";. QueryFolder Customer AccountNumber 123456
```

![Error 1](/static/odwek/Error1.png)

# Error #1
Apparently the ODWEK installer isn't smart enough to add itself to your path. Go ahead and do that.
You will need to restart your command line at this point.

![Error 1 Fix](/static/odwek/Error1Fix.png)

# AND AGAIN!
![Error 2](/static/odwek/Error2.png)

This time it loaded ars3wapi32, but there were MORE dependent assemblies that it couldn't find because they're not on the path. Depending on your architecture, you either need to add

**C:\Program Files\ibm\OnDemand Web Enablement Kit\V9.0\lib32**
or
**C:\Program Files\ibm\OnDemand Web Enablement Kit\V9.0\lib64**

I'm running a 32bit JDK on a 64 bit machine...so I went for lib32. Seemed to work. Now I can start working on my servlet.

![Error 2 Fix](/static/odwek/Error2Fix.png)

![Finally Works](/static/odwek/Works.png)

Success - it returned 0 records 🙂

# Servlet Time!
Fairly recently I've been doing some work with Spring Boot. It was fairly natural for me to use this because I'm familiar with it. I'm sure you could use any of a number of servlet technologies.

I don't want to go too deep into Spring Boot and Maven, so I'm just going to hit the big points here.

1.  I wrote a single Controller called `APIController` that has two routes.
    * /query – this queries a particular folder
    * /document – returns a document as a byte array

    One strange thing was that my document ids included non-printable unicode, so I simply replace these with a pipe before returning my JSON. There’s likely a better solution here, but it’s good enough to work for my purposes.

2. Since I’m using the ODAPI.jar, I need it on my classpath and to be included in my package JAR/WAR file. The best way to do this, is to install it into your maven.

    * Run this from the command line with Maven (<https://stackoverflow.com/a/4955695>)

    ```bash
    mvn install:install-file -Dfile=ODApi.jar -DgroupId=ODApi -DartifactId=ODApi -Dversion=1.0 -Dpackaging=jar -DgeneratePom=true
    ```
    ![Maven](/static/odwek/Maven.png)

3. I modified my Spring Boot app to generate a WAR instead of a fat JAR. There are instructions online for how to do this. It mostly involves changing your @SpringBootApplication class and some settings in POM.xml.


Note: I noticed that the initial negotiation for logging on to the ODServer was taking quite a bit of time. Too long to be performant for my users, so I cached a single connection and then simply synchronized all my methods that use it. If you have a bit of volume, more than a handful of users, it would probably make sense to implement a connection pool. I'll leave that as an exercise for the reader!

Testing this is easy enough. Fire up spring boot and issue a query via web-browser.

![Test](/static/odwek/Testinbrowser.png)

Note: I'm not handling AFP or application/line inside the servlet. This servlet is actually my middle tier, so I'm simply passing it along to my calling service (C#). I have some C# code that is parsing out the application/line format.

P.S. if you do try to parse out the application/line and it's coming from the iseries, don't forget it's in **EBCDIC**!

## Happy Coding!

Proper Date Formatting with Thymeleaf and Spring


In Spring, if you attempt to display a Date from your viewmodel in an HTML5 date input, it won't format correctly.

To fix - you can implement custom binder on your controller. Put the following at the top of your controller.
```java
@InitBinder
public void initBinder ( WebDataBinder binder )
{    	 
    SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd");
    dateFormat.setLenient(false);
    binder.registerCustomEditor(Date.class, new CustomDateEditor(dateFormat, true));
}
```

.NET Core Message Pump


I'm writing a cross-platform mouse without borders clone right now. Basically a .NET version of Synergy. 

**Tech stack: .NET Core, SignalR**

I ran into an issue with my windows hooks not working when running inside a .NET Core console. This is because if you don't include `Application.Run(new ApplicationContext())`, there is no Windows message pump. Without a message pump, your hooks won't receive anything.

Fortunately, you can implement your own message pump very easily.

At the bottom of your console app, you need to implement a message pump by calling into the native windows API function: GetMessage, TranslateMessage, and DispatchMessage.

```C#
public struct WinMsg
{
    public IntPtr hwnd;
    public uint message;
    public IntPtr wParam;
    public IntPtr lParam;
    public uint time;
    public WinPoint pt;
}

public static class WinNative
{
    [DllImport("user32.dll")]
    public static extern int GetMessage(out WinMsg lpMsg, IntPtr hWnd, uint wMsgFilterMin,uint wMsgFilterMax);

    [DllImport("user32.dll")]
    public static extern bool TranslateMessage([In] ref WinMsg lpMsg);

    [DllImport("user32.dll")]
    public static extern IntPtr DispatchMessage([In] ref WinMsg lpmsg);
}
```

And then in at the bottom of you console app - write the pump

```C#
WinMsg msg;            
while (WinNative.GetMessage(out msg, IntPtr.Zero, 0, 0) > 0)
{
    WinNative.TranslateMessage(ref msg);
    WinNative.DispatchMessage(ref msg);
}
```

Bash script to configure raspberry pi for kiosk mode


I wrote myself a script to provision the raspberry pi for kiosk mode. 

Be sure to change the password it sets.
Run it with an argument for the URL that it opens.

In the script I do some stuff with grep and sed to allow it to be run multiple times. I also explicitly set the keyboard to en-us and enable SSH for management.


## Run like this
`sudo ./setup.sh http://???`

```bash
#!/bin/bash
if [[ $UID != 0 ]]; then
    echo "Please run this script with sudo:"
    echo "sudo $0 $*"
    exit 1
fi
echo 'pi:r@aspberry' | chpasswd
update-rc.d ssh enable && invoke-rc.d ssh start
sed -i /etc/default/keyboard -e "s/^XKBLAYOUT.*/XKBLAYOUT=\"en\"/"
dpkg-reconfigure -f noninteractive keyboard-configuration

apt-get update -y
apt-get install x11-xserver-utils unclutter -y
grep -q -F '@chromium-browser --kiosk --no-first-run --disable-java --incognito' /home/pi/.config/lxsession/LXDE-pi/autostart || echo '@chromium-browser --kiosk --no-first-run --disable-java --incognito http://michaelwda.com' >> /home/pi/.config/lxsession/LXDE-pi/autostart

grep -q -F '@xset s noblank' /home/pi/.config/lxsession/LXDE-pi/autostart || echo '@xset s noblank' >> /home/pi/.config/lxsession/LXDE-pi/autostart
grep -q -F '@xset s off' /home/pi/.config/lxsession/LXDE-pi/autostart || echo '@xset s off' >> /home/pi/.config/lxsession/LXDE-pi/autostart
grep -q -F '@xset -dpms' /home/pi/.config/lxsession/LXDE-pi/autostart || echo '@xset -dpms' >> /home/pi/.config/lxsession/LXDE-pi/autostart
grep -q -F '@unclutter -idle 0.1 -root' /home/pi/.config/lxsession/LXDE-pi/autostart || echo '@unclutter -idle 0.1 -root' >> /home/pi/.config/lxsession/LXDE-pi/autostart

sed -i .config/lxsession/LXDE-pi/autostart -e "s|^@chromium-browser.*|@chromium-browser --kiosk --no-first-run --disable-java --incognito $1|"
#reboot
```

Native OSX calls using .NET Core and pinvoke


As an experiment I wanted to write some code in .NET Core that would call native OSX call to perform an action not exposed by .NET Core. I decided to make a program that would simply move the mouse a little bit.

This ended up being really simple once I figured out a few gotchas. 

- The DllImports need to reference the CoreGraphics framework, not a dylib file.
- OSX wants 64 bit types!

```C#
using System;
using System.Drawing;
using System.Runtime.InteropServices;
using System.Threading;


namespace consoletest
{
    class Program
    {
        static void Main(string[] args)
        {
            uint myDisplay = CGMainDisplayID();
            Console.WriteLine(myDisplay);
            for (double i = 0; i < 1000;i+=10)
            {
                var err=CGDisplayMoveCursorToPoint(myDisplay,new CGPoint() { X = i, Y = i });
                Thread.Sleep(10);
            }

        }


        [DllImport("/System/Library/Frameworks/CoreGraphics.framework/CoreGraphics")]
        public static extern int CGDisplayMoveCursorToPoint(uint display, CGPoint point);

        [DllImport("/System/Library/Frameworks/CoreGraphics.framework/CoreGraphics")]
        public static extern uint CGMainDisplayID();

        [DllImport("/System/Library/Frameworks/CoreGraphics.framework/CoreGraphics")]
        public static extern int CGWarpMouseCursorPosition(CGPoint point);

    }

    //osx using 64bit types, so we can't pass the .NET PointF type because it's using 32bit Single for x,y
    public struct CGPoint{
        public double X { get; set; }
        public double Y { get; set; }
    }
}
```

I tried using PointF at first and it eventually dawned on me that it was using 32bit Singles and I needed Doubles. After marshaling it kept putting my cursor at 0,0 🙂 

I think it would be pretty fun to write a Mouse Without Borders clone using .NET Core 2.1 and maybe SignalR? 

Raspberry Pi for TV Slideshow


# NOTE: I've also wirrten a script that can perform most of the setup. See here: [Bash Script](rpi_bash)


Within the past year, I've had several customers ask me for digital signage solutions. I looked at commercial solutions for them, but they either 

- didn't have all the features they wanted or 
- were out of budget.


The main features the customers needed were

- HDMI - can be displayed on normal TVs bought from wherever
- Dynamic - information being displayed needs to be pulled from either internal systems or from user-defined input (WYSIWYG)
- Possibly have slideshows
- Cheap - Obviously there's development costs, but they don't want to spend $10k for a license of something and then have to pay maintenance or whatever else
- Ability to work across the wireless network

# The plan
After doing some research, thinking, and maybe some beers - I decided I would use a Raspberry Pi running linux that boots into Chrome Kiosk mode and loads a webpage. This webpage would then display whatever I wanted using HTML5/Javascript. 

# Hardware and Software
### Raspberry Pi
​I went to Amazon and bought a Raspberry Pi 3 Kit that includes the case and memory card. Here's the one I bought: [CanaKit Raspberry Pi 3 Complete Starter Kit - 32 GB](https://www.amazon.com/gp/product/B01C6Q2GSY)

### Raspbian Desktop - https://www.raspberrypi.org/downloads/raspbian/
Make sure you download the full desktop version, as we’ll need to be launching Chrome in kiosk mode.
![Raspbian](/static/rpi/r.png)

### Win32DiskImage - https://sourceforge.net/projects/win32diskimager/

I’m not a big fan of installers, so I went and downloaded the -Binary.Zip file from the archive. This will let you run the application without having to install anything.
https://sourceforge.net/projects/win32diskimager/files/Archive/

### SDFormatter - https://www.sdcard.org/downloads/formatter_4/
This is required for formatting as FAT

# Installing Raspbian
This is for directly installing Raspbian. You could probably achieve the same using NOOBS. If you were going to install a bunch of these devices, it would probably make sense to engineer a custom image, or possible use https://www.raspberrypi.org/blog/pibakery/

1. Run SDFormatter and format your SD Card. This may not be strictly necessary, but it doesn’t hurt.

    ![r-1](/static/rpi/r-1.png)

2. Extract the Raspbian image. I use 7zip.

    ![](/static/rpi/7zip.png)

3. Run Win32DiskImage

4. Select the Raspbian image file you downloaded.

5. Select the drive of your SD card in the ‘Device’ dropdown. Make sure you chose the correct one. Otherwise, you risk damaging the data on your hard drive.
6. Select **Write** and wait for the process to finish. That’s it!
7. Now you can plug the SD card into your Raspberry Pi’s slot.

     ![](/static/rpi/20171011_191241-300x169.jpg)

# Setup Raspbian
1. Raspbian will boot automatically to the desktop.
2. Hook up a keyboard / mouse to interact
3. If you wish to set up wifi, you can do it using the GUI

![](/static/rpi/20171011_191811-300x169.jpg)

# Update Packages

Open a terminal and type
```bash
sudo apt-get update && sudo apt-get upgrade -y
```

# Install Chromium

Open a terminal and type
```bash
sudo apt-get install chromium-browser
```

*It will likely already be the latest version*

# Fix the keyboard – For whatever reason, my keyboard was set to GB.

Edit /etc/default/keyboard and change it from gb to us

Run: sudo reboot

# Install additional tools

Open a terminal and type
```bash
sudo apt-get install x11-xserver-utils unclutter
```

# Script the browser to start in kiosk mode

Run: `cd .config/lxsession/LXDE-pi/`

Run: `sudo nano autostart`

Add this line to the end
```bash
@chromium-browser –kiosk –no-first-run –disable-java –incognito https://yoururlhere
@xset s noblank
@xset s off
@xset -dpms
@unclutter -idle 0.1 -root
```

These lines will autostart the chromium browser and also prevent the pi from going to sleep.

# Set up a static IP

Run: `sudo nano /etc/dhcpcd.conf`

Type in the following lines on the top of the file: `interface eth0 static ip_address=192.168.1.XX/24 static routers=192.168.1.1 static domain_name_servers=192.168.1.1`.

Run: `sudo reboot`

# Enable SSH

In terminal type: `sudo raspi-config`

First step should be to change your user password which should be option 2. The default username is pi and the default password is raspberry.

Once you’ve changed your password head to option 8 – Advanced Options and then option A4 – SSH and just hit enabled.

SSH is now enabled, lets test it out. Head to another computer on the same network, if you are on a mac/linux computer you can do this next step from the terminal if you are a PC user however you will need to install a SSH program, my favorite being putty.

**Note:** These menu options have been moved around, just navigate around until you find them 🙂

# Wrapping up
At this point, you should have a fully-functional raspberry pi, with SSH, that you can configure to point to a website.

![](/static/rpi/20170911_161152.jpg)

Next steps are to code a custom web app that dynamically displays data from a database/internal systems. I'll get into that in Part 2!

Late Adopter

I think being a late adopter has way more advantages that being on the cutting edge. As a consultant writing code for others, I stick to old and proven tech. I feel like this confers quite a few advantages. 

1. Stable. Early adopters have ironed out the bugs.
2. Support. There will likely be extensive documentation and stackoverflow question history to guide you.
3. Social Proof. What easier way to sell a customer on using technology X, when you can point to large established companies using it too?

This isn't to say that you shouldn't use new and innovative technologies. If I was a salaried employee at a company whose primary focus was software, I would absolutely jump on some new technology or even write some myself. If your software isn't the product, but rather the tool to enable efficiency for specific business use-cases, you should probably being using .NET or Java. 

I spend a lot of time streamlining / automating business processes. Internal tablet apps, web-portals, interacting with legacy iSeries codebases, on and on. The app needs to just work 100% of the time with no intervention from you. Trust me, you don't want to feel that dread in the pit of your stomach every time your phone rings. 

# Things I've Completely Avoided Because I'm Busy Shipping

1. Angular 1.x 
2. The entire javascript ecosystem outside of jquery
    * I mean I've of course made a chatroom with node, express, and socketio
3. Silverlight 
4. Most of J​ava except Spring.
5. Blockchain - hard pass on all of this.

# Things I've Avoided But I'm going to Deep-dive On In 2018

1. React
2. .NET Core
3. Modern CSS
4. Docker

2017 Recap

Someone on twitter said that doing a yearly retrospective is extremely valuable - so here I am. I've decided one of my 2018 goals will be to live a more public-facing life, partially for professional reasons, and this is a good first step.

# On the road
​2017 has been a wild-ride. I moved to Cary in December 2016 and have spent a whole year here at this point. Most of my client-base is still in Winston, so I've been doing a bit of driving this year. I didn't drive every day, but it probably averages out to about one trip a week. 12,300 miles total. Travel time based on google maps: 227 hours. **Wow.**  That's  5 1/2 weeks of time that I could have spent in a much more productive way, or at least in a more enjoyable way. What if I just flat out refused to travel and told everyone I needed to be remote or no-dice? Would certainly be better for my mental health...

## Raleigh
I love being back in Cary/Raleigh. Most of my close friends are still in the area and we are finally able to do things together again. My wife and I have started regularly attending / hosting dinners and going to events with our friends. I don't think it's a stretch to say that both me and my wife are introverts, so making new friends in Winston never came naturally. It's nice to be able to rejoin an old established group.

## Goals Accomplished

- Got a fence built.
- New puppy (Belle).
- House decorated, furniture purchased, home office set up.
- Expecting first child early 2018.
- Cultivated an ongoing relationship with a really good Raleigh client for 2-days a week of work.
- Shipped a LOT of code. 4-5 new web applications into production. Ongoing maintenance work and enhancements.
- Fully scheduled myself moving into 2018.
- Made 20% more money that I did in 2016.
- Consolidated all of my todo lists into Trello (using GTD style lists).
- Wrote a few blogs posts.
- Got back into Twitter. This is something I've been wanting to do.

## Shortcomings

- I traveled way too much.
- I attempted to offload some projects of mine to 1099 and it didn't work out. I feel like I can probably pin most of the problems here on myself (communication and setting expectations). I'm likely going to give this another go - will write about it.
- Had a lot of trouble with time-management. Multiple last-minute marathon code sessions to meet deadlines.
    * Side Note: these were all deadlines that I basically set myself by scheduling meetings promising to show the latest changes. If I don't do stuff like this, I never feel the pressure and don't get things done. Perhaps I should set more realistic goals and tell people "NO" more. 
- Didn't blog as much as I should have.
- Had a lot of trouble leaving work at work. Consolidating my todo lists has really helped here.

# 2018 Goals
I'm excited to see what 2018 brings. Here's a short list of goals and some ideas on how I'm going to accomplish them.

- Less travel - the upcoming arrival of our baby girl has made this one really easy for me. I told all my clients, truthfully, that I don't want to travel very far when my wife is in here 3rd trimester. I am also not going to be travelling after she's here because my wife needs me home. My Raleigh client has me onsite 2 to 3 days a week in 2018, and the rest of the time will be spent remote. 
- Better time management - I cannot realistically continue to do late-night marathon code sessions. It's just not sustainable. I am planning to be very disciplined with my time-management in 2018. I'm also considering ways I can be held accountable when working from home on open-ended projects. Maybe I'll start live streaming myself in the office working? 
- Less Coffee - I've gone a little overboard on the coffee and really need to cut back. In January I'm going to begin my journey.
- Make 20% more, again. This will be interesting. I'm close to hitting a hard-cap based purely on the fact that there's only so many hours in the day. Even with the reduced travel, I will need to scale in other ways to accomplish this. I'm thinking productized services and possibly hiring another developer. 
- Work on the business rather than in the business. e-myth stuff.
- Spend time with my wife and new baby. The only way I can achieve this in the way I want is to have extreme discipline with my time-management. ​
- Continue learning. This year I want to deep-dive on .NET Core, CSS grid, and React.
- Launch something. I want to launch something public-facing of my own. I do it for clients all the time, I should do something for myself. 


How to batch change the date received on Office 365 mail

Sometimes bad things happen to good mailboxes. We were recently doing a fairly standard mailbox migration from Google Apps to Office 365. This mailbox in particular was about 30 GB and had mail going back 20+ years. It also had some previous migration history.

# The History
Nothing too unusual here. In 2014 the mailbox was migrated off of Exchange server and onto Google Apps. Now, in 2017, it was being migrated from Google Apps to Office 365.

Everything went perfectly, but there was one problem. A large portion of the mail was showing an incorrect date received. During the initial migration, Google Apps had set the Received Date to be the date of the migration on a large portion of messages.

# Fixing it
We added the account in question to one of our developers' laptops. Take care to sync ALL of the mail.

![Change Account](/static/redemption/redemption1.png)

Once all the mail is pulled down from the server, you can inspect the messages using a tool called MFC MAPI

Here's a look at an example message.

![Message](/static/redemption/redemption2.png)

This is interesting - but it's just not feasible to fix 20 years of mail by editing each message. Enter Redemption.

# Redemption
[Redemption](http://www.dimastr.com/redemption/home.htm) is a COM library that uses extended mapi to duplicate the functionality of the Outlook Object Model. Basically, we can manipulate outlook using a program. Now we're in business.

Sometimes, you just need to write a quick and dirty program, run it once, and throw it away. That's exactly what we did.

In this particular case, we noticed that the Sent Date on the effected mail was still accurate. So, we just iterated across every single message from before 06/2014, and set the Received Date to match the Sent Date.

[Full Project on Github](https://github.com/michaelwda/MAPIReader)

```C#
class Program
{
    static void Main(string[] args)
    {
        //tell the app where the 32 and 64 bit dlls are located
        //by default, they are assumed to be in the same folder as the current assembly and be named
        //Redemption.dll and Redemption64.dll.
        //In that case, you do not need to set the two properties below
        RedemptionLoader.DllLocation64Bit = @"Redemption/redemption64.dll";
        RedemptionLoader.DllLocation32Bit = @"Redemption/redemption.dll";
        //Create a Redemption object and use it
        RDOSession session = RedemptionLoader.new_RDOSession();

        session.Logon(Missing.Value, Missing.Value, Missing.Value, Missing.Value, Missing.Value, Missing.Value);
        
        var stores = session.Stores;
        foreach (RDOStore rdoStore in stores)
        {
            if (rdoStore.Name.Contains("michael"))
            {
                var folderId = rdoStore.RootFolder.EntryID;
                PerformMailFix(folderId, session);
            }

        }
        Console.WriteLine(count);
        Console.ReadKey();
        session.Logoff();
        
    }

    private static int count = 0;
    private static void PerformMailFix(string folderId, RDOSession session)
    {
        var folder = session.GetFolderFromID(folderId);

        if (folder.FolderKind == rdoFolderKind.fkSearch)
            return;

        var before=new DateTime(2014,06,30);
        foreach (RDOMail item in folder.Items)
        {
            if (item.ReceivedTime >= before) continue;
            var diff = item.ReceivedTime - item.SentOn;
            if (!(diff.TotalHours > 10)) continue;
            Console.WriteLine("" + item.Subject + " - " + item.ReceivedTime + "    " + item.SentOn);
            count++;
            item.ReceivedTime = item.SentOn;
            item.Save();
        }

        Console.WriteLine(folder.DefaultMessageClass);
        //do the same fix for all subfolders
        foreach (RDOFolder subFolder in folder.Folders)
        {
            PerformMailFix(subFolder.EntryID, session);
        }
        
    }
}
```

# Syncing Up
Once we developed and ran the program, we launched Outlook on our development laptop and let it sync up to the Office 365 cloud. Once the sync was completed, all other connected devices pulled down the changes automatically and showed the correct date!

Split And Encrypt PDF

# Why
At my company, we recently had a need internally where we wanted to split and subsequently encrypt each page of a PDF. In our exact case, we wanted to split a payroll PDF file into individual paystubs based upon the employee's name. Once we had each page separated, we also wanted to encrypt the employee's paystub using a password that they provided.

[Check it out on github](https://github.com/michaelwda/PayrollSplitter)


# How to do it

​Use a PDF library to open the file in question.
Find a way to extract the text from each page, and look for a key-phrase or unique id.
Separate out that single page, encrypt it, and save it using the aforementioned PDF library.

# PDF library
I originally developed this application using PDFSharp, but a change in PDF format made the application stop working. This ended up being due to PDF iref streams, which are unsupported in the current version of PDFSharp (they *are* supported in the current beta). I switched to a library called [iTextSharp](https://github.com/itext/itextsharp) that got the job done. Please note,  iText is AGPL and this can't be used commercially without purchasing the rights from iText.

# Settings
There's a settings section where you can define our employees, along with a unique identifier. This unique identifier is used to figure out of a paystub actually belongs to them. Depending on the size of your company, you could probably just use the employee's full name.
![ss1](/static/paystub/ss1.png)

# Loading the PDF
On the main screen, you can just point ourselves to the PDF in question. At this point, the program will process the PDF, split the pages, and encrypt the resulting files. The preview button will allow us to take a peek. When you attempt to open the preview PDF, you can see that the file is already password-protected.

![ss2](/static/paystub/ss2.png)

![ss3](/static/paystub/ss3.png)

![ss4](/static/paystub/ss4.png)

# Save all stubs to a folder
Once you have confirmed that everything is looking good, save all of the encrypted files off to a directory.

![ss6](/static/paystub/ss6.png)

![ss7](/static/paystub/ss7.png)

# The Code
The real meat of all of this is inside the payroll splitter service. You can see the entire file on github.

Text-extraction is done using the iTextSharp simple text extraction.

Then simply use PdfCopy, a memory stream, and PdfStamper, to seperate/encrypt the individual pages.

```C#
public List<PayStub> GenerateIndividualStubs(string filename)
{

    List<PayStub> stubs=new List<PayStub>();
    using (var reader = new PdfReader(filename))
    {
        
        for (var page = 1; page <= reader.NumberOfPages; page++)
        {
            //extract page text
            var strategy = new iTextSharp.text.pdf.parser.SimpleTextExtractionStrategy();
            var currentText = iTextSharp.text.pdf.parser.PdfTextExtractor.GetTextFromPage(reader, page, strategy);
            
            var paystub = FindPayStubMatch(currentText);
            if (paystub == null) continue;

            //write this individual page into a temp file
            var outFileName = System.IO.Path.GetTempPath() + Guid.NewGuid() + ".pdf";

            var pdfDoc = new Document(reader.GetPageSizeWithRotation(page));
            var stream = new MemoryStream();
            var pdf = new PdfCopy(pdfDoc, stream);
            pdf.CloseStream = false;
            pdfDoc.Open();
            pdf.AddPage(pdf.GetImportedPage(reader, page));                    
            pdf.Close();
            pdfDoc.Close();
            stream.Position = 0;
            
            var stamper = new PdfStamper(new PdfReader(stream), new FileStream(outFileName,FileMode.Create));
            if ((paystub.Password + "").Length > 0)
            {
                stamper.SetEncryption(Encoding.ASCII.GetBytes(paystub.Password), Encoding.ASCII.GetBytes(paystub.Password), PdfWriter.ALLOW_PRINTING,
                    PdfWriter.ENCRYPTION_AES_128 | PdfWriter.DO_NOT_ENCRYPT_METADATA);
            }
            stamper.Close();
            stream.Close();

            paystub.Filename = outFileName;                
            stubs.Add(paystub);

        }


    }
    return stubs;
}
```

# Next Steps
The logical next step here would be to tie this into an email server. I've implemented something similar to this for several clients wanting to automatically send through their exchange server.

1. We pull the employee list out of their existing accounting solution automatically using Sql. This reduces the burden of maintaining two systems.
2. Instead of saving the PDFs to a folder, we automatically send them out through an email server directly to the employee's email with the PDF as an attachment.

C# replacement of ODWEK applet

# NOTE: I've recently coded this up the "correct" way using the Java Api. See here: [ODWEK Middleware with Spring Boot](ODWEK_middleware_springboot)

## NPAPI
Google Chrome completely removed NPAPI as of September 2015. http://www.chromium.org/developers/npapi-deprecation

I think most people would agree that this is a *good* thing for security and web-standards. Remember when Apple said NO to Adobe Flash on the iPhone and everybody went crazy? I feel like history has proven that decision the be correct, and this is just an extension of that.

Anyways, google provided plenty of notice that they were going to do this. They even deprecated the feature, but left a way to turn it back on for a short time. A smart developer would have seen this and been hard at work implementing a new solution that doesn't rely on NPAPI. Apparently, I am not smart.

As a consultant, I'm not really in the seat most of the time. I was using an IBM applet that relied on NPAPI it and it worked, so everything was good. I eventually got the call that the ODWEK viewer wouldn't load and they couldn't view receipts. The client had to go back to using the desktop client instead of being able to click the link from the CSR portal I had written. Sure, they could just switch to IE, but I decided to go a different route.

## ODWEK
For the unitiated, ODWEK stands for On-Demand Web Enablement Kit. Basically a web api for viewing data stored in IBM's content manager. Think iSeries. The viewer itself launched as an applet in the browser (NPAPI) and would pull / display the content. For this client, all of their content was TIFF, PDF, HTML, or TXT. Nothing exotic.

I stepped back for a moment on this and thought about it. The applet is code, running locally. It MUST be pulling data across the network and then just displaying it. Why don't I just find out how it's doing that, perform the operation server-side, and then spit the content out to the browser with the correct MIME type.

## Digging in

The exact workflow was:

1. Hit arswww.cgi for a listing of receipts/images
2. Click the link, launching the viewer. Tiffs and PDFs seemed to just spit out to the browser with no viewer. The payment receipts folder was attempting the load the applet unsuccessfully.

So, I need to write a wrapper around arswww.cgi, intercept all the HREF elements, and call into my own code. Cool.
What I really needed to find out, first of all, was if I could even reverse engineer the web calls that the applet makes?

I grabbed a copy of the applet off of the iSeries and decompiled it using jd-gui. I'll skip my journey of trying to grok the decompiled source, but the relevant code was in a file called LDVServerInterface.class.
![Decompiled](/static/odwek/Decompiled.png)

The basics of this are:

1. There will be a series of parameters in the format of "param=n". After the param, there will be N bytes of data.
2. If the parameter is a physical or logical buffer, read it into a byte stream.
3. If the parameter says "DATA", read two more bytes into the bytestream and then deflate it into a string value. This is your data.
```C#
private string GetReceipt(string requestUrl)
{
    string data = "";
    var client = new WebClient();
    var byteStream = client.OpenRead(requestUrl);

    //var table = new Dictionary<string, string>();

    var builder = new List<byte>();
    var b = 0;
    while (byteStream != null && (b != -1 && byteStream.CanRead))
    {
        //read param
        for (b = byteStream.ReadByte(); (b != -1) && (b != 61) && (b != 10); b = byteStream.ReadByte())
        {
            builder.Add((byte)b);
        }

        if (builder.Count != 0 && b == 61 && b != -1)
        {
            var param = Encoding.UTF8.GetString(builder.ToArray());
            var valueBuilder = new List<byte>();
            //read value
            while ((b = byteStream.ReadByte()) != 10)
            {
                if (b == -1)
                    break;
                valueBuilder.Add((byte)b);


            }
            var paramValue = Encoding.UTF8.GetString(valueBuilder.ToArray());

            if (param == "DATA")
            {
                var readLength = int.Parse(paramValue);
                byteStream.ReadByte();
                byteStream.ReadByte();
                using (var deflate = new DeflateStream(byteStream, CompressionMode.Decompress))
                {
                    using (var reader = new StreamReader(deflate, true))
                    {
                        data = reader.ReadToEnd();
                    }
                }

            }
            else if (param == "PHYSICAL_BUFFER" || param == "LOGICAL_BUFFER")
            {
                var n = int.Parse(paramValue);
                var localByte = new byte[n];
                var i1 = byteStream.Read(localByte, 0, n);
                while ((i1 != n) && (i1 != -1))
                {
                    i1 += byteStream.Read(localByte, i1, n - i1);
                }

                byteStream.ReadByte();

            }
            else
            {
                //table.Add(param, paramValue);
            }
        }

        builder = new List<byte>();

    }

    byteStream.Close();
    return data;
}
```
 
## Wrapping it
Next steps here:

1. Write a wrapper around arswww.cgi that parses the HTML, modifies any HREF elements, and points it back internally to my own file intercept code.
2. Feel bad that i'm using arswww.cgi, because it too, is deprecated....

So, my wrapper page. Basically just a plain aspx with a literal tag.
`<asp:Literal ID="ScrapedData" runat="server"></asp:Literal>`

Code-behind intercepts some parameters and sends them along. Then I parse the HTMl using HTMLAgilityPack and replace the Href of all the links.
```C#
var account = "" + Request["AccountNumber"];
var folder = "" + Request["Folder"];

var query = "Account Number";

if (folder == "PaymentReceipts")
    query = "AccountNumber";

if (folder == "")
    folder = "Customer";

string html;
using (var client = new CookieAwareWebClient()) // WebClient class inherits IDisposable
{
    html = client.DownloadString(string.Format("http://10.1.1.1/scripts/arswww.cgi?_f={0}&_a=d&{1}={2}&_server=???&_user=???&_password=???&_function=logon",folder,query, account));
    Session["CookieContainer"] = client.CookieContainer;
}

var doc=new HtmlDocument();
doc.LoadHtml(html);

var tables = doc.DocumentNode.SelectNodes("//table");
if (tables == null) return;

//modify all the links in table
var links = doc.DocumentNode.SelectNodes("//a");
if (links != null)
{
    foreach (var link in links)
    {
        var href = link.Attributes["href"].Value;
        var encodedHref = HttpUtility.UrlEncode(href);
        link.Attributes["href"].Value = "OnDemandFileIntercept.aspx?AccountNumber=" + account + "&url=" +
                                        encodedHref;
    }
}

//output table
foreach (var table in tables)
{
    ScrapedData.Text += table.OuterHtml;
}
```
 
Then my intercept does it's thing.

```C#
protected void Page_Load(object sender, EventArgs e)
{
    var url = "" + Request["url"];
    var cookies = Session["CookieContainer"];
    var account = "" + Request["AccountNumber"];
    if (cookies == null || url == "") return;

    var decode = HttpUtility.UrlDecode(url);
    if (decode.Contains("_f=PaymentReceipts"))
    {
        var requestUrl = "http://10.1.1.1" + HttpUtility.UrlDecode(url) + "&_applet=1&_line=native&_s=???&_u=???&_p=???";
        var receiptData = GetReceipt(requestUrl);

        receiptData = receiptData.Replace("\n", "<br>");
        Response.Write(receiptData);
        Response.ContentType = "text/html";


    }
    else
    {
        using (var client = new CookieAwareWebClient()) // WebClient class inherits IDisposable
        {
            client.CookieContainer = cookies as CookieContainer;
            var data = client.DownloadData("http://10.1.1.1" + url);

            Response.ClearContent();
            Response.Clear();
            Response.ClearHeaders();

            var mime = GetMimeFromBytes(data);

            Response.ContentType = mime;

            if(mime.Contains("application"))
                Response.AppendHeader("content-disposition", "filename=" + account + ".afp");
            else
                Response.AppendHeader("content-disposition", "filename=" + account + ".tiff");

            Response.OutputStream.Write(data, 0, data.Length);
        }
    }

    
}
```
 
Nice!
So, at this point I just deployed the code and changed the link on the customer CSR to point to my ODWEK interceptor. Problem solved...for now 🙂

Michael Davis