MVP Award Day – The July cycle!

fightpaw

The first rule of Fight Claw Club is…

By the time this post is finally published, many of us will know the results of whether our renewal (or new award status) has been successful (or not). While I have only been the recipient of the award since Jan 1st, 2016 I had previously been nominated on many many occasions yearly since 2011, so I almost feel like a long-term veteran of the award cycles, and only know too well how rejection feels – so if you are one of those today, trust me when I say “You are not alone – I understand your pain”.

Program changes

When the changes to the award program renewal cycles (to move almost everyone to a July 1st renewal) were announced in now what seems like an eternity ago (perhaps over 15-18 months ago now), I confess I had mixed feelings about such a change. On one hand it gave some MVPs more time to get their “house in order” for an extra 3-6 months longer (assuming they needed to “up” their contributions) due to their moving to a new renewal cycle, but on the other hand, it meant an awful lot of MVPs would go into the same bucket for consideration, thereby potentially increasing the risk of you being overlooked. One observation I had (and I’m not entirely sure if this is correct or not) was that post cycle change, the numbers of awardees increased significantly month on month. And if that is the case, it could have contributed to raising the “bar” for renewal this time around.

Renewal time

Over the past couple of weeks there have been some high profile casualties announced privately, including several who I consider being the epitome of what an MVP is all about, and others who openly admitted that their priorities have changed and not had the time, resources, or inclination to keep up with their contributions over the last 12 months. For the former category, I truly cannot express how gutted I am for you and hope that you make it back onto the program as soon as possible. For those that haven’t hit the bar, then I’m sure you know what to do to make it back.

What next?

For those that have missed out on their renewal, I am happy that Microsoft has the new MVP Reconnect program in place, which I think is an excellent way for people not renewed to stay connected. I would like to see this program elevated in importance over the coming years -and especially for it to be easier for current MVPs to communicate with rMVPs. That way, missing out on renewal will feel less like you are losing lots of friends and professional contacts and more like hanging out in a different forum, with an easier (potential) route back should you increase your community contributions.

I would also like to see a similar program for potential MVPs (people who are almost “there”), which would allow Microsoft to substantially increase their pool of talent, but without the cost overhead that running the MVP program incurs. This would provide an initial route into the full program and allow people to be less overwhelmed, understand what is expected to become an MVP, make them better prepared, and more useful as Microsoft evangelists (should the MVP email finally land in their inbox). Given the sheer amount of talent out in the community, I see this as a huge win/win for Microsoft and community contributors. Perhaps Microsoft could initially adopt and formalize their various “Advisor programs” into an MVP stepping stone?

Final thoughts

I’ve been thinking a lot over the last few months about what I’m actually specifically delivering to the MVP Program, and the general community at large. Whilst it is fair to say I am really happy with the level of community contributions I make (the unhappy wife and kids are a fairly good barometer that I am investing *a lot* -perhaps too much of my time in this area), I do think I can (and need) to add more value to Microsoft. In this last 12 months, a couple of particular stand-outs would probably be my SQLSaturday and SharePoint Saturday events (in which we always have lots of early adopter sessions – which actively promote the use of such technology), delivering a Microsoft and GDPR session at short notice to Future Decoded (which obviously pushed on-premises and Cloud solutions in a very hot topic), and feedback to product team regarding SQL Server on Linux (my motivations were also helped having been working as the Technical Editor for the book of the same name). However, there are lots of other areas I either need to improve or to add to my list. Those specific things are (in no particular order):

  1. Focus my energies towards learning and presenting on several emerging Azure technologies.
  2. Diversify even more (i.e. start investing more time in Business Intelligence and Machine Learning).
  3. Get involved in the Microsoft Docs program (where possible) to improve the quality of MS Documentation.
  4. Increase my blog output (for it has dropped somewhat of late) – and also try to blog on things I learn in 1 and 2!

There are of course a million and one other things that I could probably do better, but for the sake of time and your patience, I will leave it there for now.

Thank you very much to everyone involved in the MVP Program, congratulations to all awardees (and re-awardees), and commiserations if you have missed out on this occasion. Your efforts are still appreciated by the community with or without those three letters so keep up the good work!

 

Posted in Certification, Community, SQLServerPedia Syndication | Tagged | 1 Comment

Reading very BIG text files using PowerShell

I frequently have to troubleshoot a variety of data feeds (populated from very very large files) which are regularly ingested into SQL Server for reporting and data validation purposes. From time to time, these feeds break due to malformed or invalid data provided by the third parties, and often due to the size of the files and limitations of third-party text file applications, loading them into a text file viewer is not possible.

I recently ran into this very same problem in which the bcp error message stated:

An error occurred while processing the file “D:\MyBigFeedFile.txt” on data row 123798766555678.

Given that the text file was far in excess of notepad++ limits (or even easy human interaction usability limits), I decided not to waste my time trying to find an editor that could cope and simply looked for a way to pull out a batch of lines from within the file programmatically to compare the good rows with the bad.

I turned to PowerShell and came across the Get-Content cmdlet to read the file, which looked like it would give me at least part of what I wanted. At this point, there are quite a few ways to look at specific lines in the file, but in order to look at the line in question and its surrounding lines I could only come up with one perfect solution. By passing the pipeline output into the Select-Object cmdlet I could use its rich skip and next functionality.

Please note that the following code examples all use PowerShell’s multiple line wrap functionality for readability purposes on this web page only. Normally I’d put all these statements on a single line (obviously minus the ` line wrapping character/s)

Basically, the following will read the file in question, skipping the first 123798766555672 lines and return the next 6 lines (I can therefore deduce that my error row will be the last line in the set with 5 good lines above):

Get-Content D:\MyBigFeedFile.txt | `
Select-Object -skip 123798766555672 -first 6

Ok. So that’s cool and works very quickly given the size of the file (in this case it was approximately 3 minutes to return!). Perhaps the final thing we would want to do is to pipe into another text file for repeat viewing (so we don’t have to keep re-reading this huge text file over and over) or perhaps you might want to use the Out-GridView cmdlet. For either option, all we need to do is direct the pipeline output now into the respective cmdlet.

Therefore the full solution to my specific problem is:

Get-Content D:\MyBigFeedFile.txt | `
Select-Object -skip 123798766555672 -first 6 | `
Out-File Output.txt

Conclusion

I think this final solution is pretty cool, is incredibly fast and works like a dream. It appears much quicker than anything else I could put together (or use out of the box) and seems to avoid any of the limitations (particularly in 32bit applications) of GUI based editors. However it is worth pointing out that I have not monitored memory consumption whilst executing this piece of code on very large files so I cannot confirm how efficient or safe this would be to do on a mission-critical server, but it is my belief that the Get-Content cmdlet is a stream operation and so should be safe. Either way you should do your own testing folks!

Posted in PowerShell, SQLServerPedia Syndication | Tagged | 2 Comments

Installing Docker on Linux Mint

Ok, so first things first. This is not a ground shaking post of revelation, and ultimately all the information you need can be found directly from Docker, but like all good posts this is intended to address any confusion or ambiguity you may find when installing Docker on Linux Mint and join all the dots for you.

A web search will almost certainly point you to lots of similar posts, mostly (if not) all of which start instructing you to add unofficial or unrecognized sources, keys etc. Therefore my intention with this post is not to replace official documentation, but to make the process as simple as possible, whilst still pointing to all the official documentation so that you can be confident you are not breaking security or other such things!

You can head over to the following Docker page Get Docker CE for Ubuntu for the initial setup and updates, but for simplicity, you can follow along below.

First run the script below in order to update your local files from configured repositories, install required packages, and add the official Docker GPG key.

# Ensure your repositories are up to date
sudo apt-get update

# Install required packages
$ sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

#Add Docker’s official GPG key:
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

#Check the GPG fingerprint successfully added (should see output from this command)
sudo apt-key fingerprint 0EBFCD88

Now that the package repository has been added, you can now install Docker Community Edition from apt as follows:

sudo apt-get update
sudo apt-get install docker-ce

Once this has been done, next up is perhaps the most important step (in terms of potential problems) -and that is adding the correct repository for your version of Linux Mint. The issue you face is that Linux Mint uses its own release codenames and so the default script (provided by Docker) picks this up rather than the (required) Ubuntu release -its the $(lsb_release -cs) piece of code in their script. Instead, you will need to find out your Mint release name and replace this with the correct Ubuntu package base.

Find out your Linux Mint short codename by running the following:

lsb_release -cs

In my case I find that I am running Linux Mint Serena. Next, you need to find out the short codename of the Ubuntu base build that your edition of Mint is derived from. To do this, visit the Linux Mint Releases page.

From this page, I can see that Serena uses the Xenial package base (as below):

So now all we need to do is add the right repository for the right package base (note I have added xenial to the script below). In your case, you may be using a new or older edition of Mint, so simply replace the word “xenial” in the script with the correct package base relevant to the version of Mint you are using.

#Add the Docker repository for the Xenial build
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   xenial \
   stable"

Once this is completed you then need to perform the Docker post-installation tasks which you can find here. These tasks are really there to prevent you having to keep running all Docker commands using the privileged sudo command. For instance, without going any further you *could* already now run the following command to list all current downloaded docker images (there should be none).

#list docker images (using privelaged mode)
sudo docker image ls

But we can avoid having to keep specifying sudo by running the following:

# Create a new docker group
sudo groupadd docker

# Add your user to the docker group.
# This script assumes that your current user
# is the one you want to be a docker admin
sudo usermod -aG docker $USER

It is now important that you log out of your session and back in, in order to pick up the new security context in your session, otherwise you may be greeted with the following text when attempting to run your docker command without sudo:

retracement@mylinuxhost ~ $ docker images
Got permission denied while trying to connect to the
Docker daemon socket at unix:///var/run/docker.sock:
Get http://%2Fvar%2Frun%2Fdocker.sock/v1.35/images/json:
dial unix /var/run/docker.sock: connect: permission denied

So if you have followed the instructions correctly, you should be able to list docker images (or any other docker command) without requiring sudo as follows:

#list docker images (non-privelaged mode)
docker image ls

And that’s it. Docker is now ready for you to run containers on your shiny Linux Mint desktop.

Posted in Docker, Linux | Tagged | Leave a comment