MyGet Blog

Package management made easier!

NAVIGATION - SEARCH

MyGet's NuGet and NPM news from the community

Many are returning from summer vacation, others have been enjoying the tranquility of summer holiday. Whichever side you’re on, we at MyGet have been watching the NuGet and NPM community news in the past few weeks. In this post, we bring you some interesting blog posts and articles, curated by our MyGet founders Xavier and Maarten. Follow @MyGetTeam on Twitter for more!

NuGet news

NuGet news, curated by MyGetOn the NuGet blog, the NuGet client 3.5 RC has been announced, with support for new target frameworks and lots of performance improvements. Additionally, the NuGet team started working on better documentation, now available as a preview on http://docspreview.nuget.org.

More from the NuGet team: they made some changes to the expiring API keys policy. At MyGet we’ve always made this opt-in, and the NuGet.org gallery will now do the same.

New to NuGet? Rohit Chopra has you covered with his article “NuGet – A Powerful way to share your code”. While focused on NuGet, it’s a nice summary of why you want to use a package manager in your projects. Xiao Ling has a step-by-step post on creating and publishing .NET Core packages.

Building things in Unity? Wondering what NuGet is? Ashley Davis has you covered with his introduction to using Unity and NuGet. The Unity solution templates don’t easily allow working with NuGet, but there are some easy workarounds. A good example is demonstrated, installing JSON.NET into a Unity project.

Have you been consuming NuGet, and just started looking into creating your own NuGet packages to share them with team mates or with the world? Learn about publishing your first .NET Core NuGet package with AppVeyor and MyGet  - Andrew Lock gives a good step-by-step tutorial on what you need in code, and how AppVeyor and MyGet can be used to build and distribute your code.

On a similar topic, Maarten Balliauw has a post on Building NuGet (.NET Core) using Atlassian Bitbucket Pipelines. Pipelines is Atalassian’s continuous integration service that runs on Docker and Linux. And since .NET Core is a first class citizen on that platform, why not use it to build and test NuGet packages?

NPM news

NPM news, curated by MyGetLet’s start on the tooling side. Node has gotten two new releases, 4.5.0 and 6.4.0. Mostly bugfixes, better profiling support and improvements in objects and function contexts for debuggers. On the npm side, there’s now 2.15.10 and 3.10.7, with improvements to how scoped dependencies are handled and several other bugfixes.

Did you know the two millionth package version was just published to npm? If you have as well, congratulations! This is a pretty epic milestone in the Node.js community.

Laurie Voss, COO at npm, has a great talk titled “Abstractions, npm past, present, future”. It covers what is npm and where it came from, where the ecosystem stands today and what the plans are for the future. Highly recommended!

New to node? Have a look at Node Hero’s blog post series! These thirteen articles cover everything from getting started with node and npm, to building a web app, security, monitoring and all other aspects of building a node application.

Npmjs.org added web hook support a while back. Julian Gruber did a proof-of-concept where updated dependencies are automatically deployed in the application. Not the best idea, given that your deployment may break because of an updated dependency, but still quite cool. Package update? Deploy!

Into the Internet of Things? One such thing is the International Space Station! Dave Johnson has a nice post Node.js IoT: Tracking the ISS through the Sky where he uses JavaScript to capture GPS coordinates from the IIS and compares it to your home location to create a real-time tracker.

We’re thinking about doing this type of post each month. Let us know if you’d like that or not, using the comments below or reach out on Twitter.

Happy packaging!

Deprecation notice: SymbolSource integration will end on November 1, 2016

On November 1, 2016, MyGet will end integration with SymbolSource.org, making our built-in symbol server the only option for symbols hosting with MyGet.

When working with NuGet feeds, symbols packages can be pushed so that consumers of the package can step through the source code and integrate with Visual Studio and tools like WinDbg. MyGet has always offered two options for handling symbols packages: using our built-in symbol server or using SymbolSource.org.

With the advent of .NET Core and native debugging on platforms like Linux and Mac OS X, we’re working closely with Microsoft on providing the best symbols-based debugging experience, an experience which we can only guarantee when using our built-in symbol server.

Please update your Visual Studio configuration and/or continuous integration servers by November 1, 2016 to make use of MyGet’s symbol server. Your feed’s “Feed Details” tab provides the correct URL’s for pushing and consuming symbols packages.

Note that the SymbolSource.org URL can still be used for consuming existing symbols packages after November 1, 2016. Account synchronization from MyGet with SymbolSource.org will end. We recommend updating your systems to make use of MyGet's built-in symbol server to ensure continuity of working with symbols packages after November 1, 2016.

Happy packaging!

Keeping feeds clean with retention rules

MyGet Package Retention Rules help clean up your NuGet npm feedMany developer teams use MyGet for storing their continuous integration and/or nightly builds of NuGet, npm, Bower and VSIX packages. As more and more packages get added, it may become harder to manage them all. Some packages may be used in projects, while others are not. Let’s go over the options available for housekeeping.

By default, MyGet keeps all package versions available on our feeds. Every package pushed is there forever, unless manually removed or removed by package retention. By setting retention rules, it is possible to automatically trim the list of packages to X latest packages, keeping into account package usage in projects and package dependency trees.

Configuring retention rules

Retention rules are defined per feed. Some feeds may have more aggressive retention rules defined, other may not have them enabled at all. From the Retention Rules, we can define:

  • the maximum number of stable versions to keep
  • the maximum number of prerelease versions to keep
  • whether to keep depended packages or not – enabling this makes sure package restores always complete successfully by keeping the dependency tree in its entirety
  • whether to allow removal of packages that have downloads – enabling this option ensures that packages that are being used in projects never get deleted

Setting retention rules

Keeping a specific package around

Retention rules are quite brute-force: they will always remove all packages that match the configured rules. Luckily, MyGet lets us “pin” packages which we want to keep around. For example, we may want to only keep the latest 10 pre-release versions while still keeping around the 20th pre-release version we’re still using in our projects.

From the package details page, we can define which package versions should never be considered by retention rules by using the Pin button next to the package.

Pinning packages so they do not get removed

We can pin package per version, or all versions at once using the button at the top of the Package History list. Of course, we can also Unpin packages using the same approach. Once a package is unpinned, retention rules are allowed to remove them.

Custom retention rules using web hooks

Using the built-in retention rules may not be enough. For example, what if we want to run retention rules based on other conditions than the latest version? What if we want to only remove packages when there is a full moon? Using web hooks, we can subscribe to certain feed events (like “package added”) and run our custom logic to optionally remove packages from our feed. We have a complete example available that helps getting started with web hooks.

Learn more about package retention in our documentation.

Happy packaging!

Improved build log viewer with error navigation

We have just deployed a newer version of our build log viewer. When using MyGet’s build services to compile and package NuGet, npm or VSIX packages, the build log viewer now has colored output as well as line numbers that have hyperlinks. Want to share a certain line in the build log with a colleague? Click the line number and send the link so they can open the build log right where you left.

By making less important build output less prominent and by highlighting more important messages, reading and analyzing the build log becomes much easier: less important messages have a gray color tone, normal messages are white. Warnings and errors are highlighted in yellow and red, making them much easier to spot.

Build log with colored output

When warnings or errors are found in a build log, MyGet now shows additional navigation buttons at the top. Next to the number of warnings or errors, the up and down arrows can be clicked to jump to the next important message in your build log.

Warning and error navigation

We’re looking forward to hearing your thoughts on this improvement. Let us know through the comments below or drop us a note via e-mail or Twitter.

Happy packaging!

MyGet by the numbers

As it is a slower time of year and many people are taking some vacation, we decided it would be a nice time to collect some numbers around MyGet. Of course we can’t share all of our statistics, but it’s always fun to look at them. MyGet has a shared tenant, www.myget.org, hosting and serving NuGet, npm, Bower and VSIX packages. And debugger symbols, of course. This shared tenant will be the focus of this post. Our MyGet Enterprise plans, which run on their own dedicated tenant infrastructure, are not included in these statistics.

Number of machines

Our web application is hosted on Microsoft Azure across three different regions. The shared tenant has its primary location in the West Europe region, and a secondary in US East. Both locations run 3 front-end web server machines and 2 back-end machines for handling anything that doesn’t have to happen real-time. These machines are all general purpose “A3” machines. We have a tertiary deployment in US West for some Enterprise customers, where a similar deployment lives.

Since we use auto-scaling based on time of day, we don’t have the same amount of machines throughout the day. On average, we run 10 machines for our front-end.

Another deployment lives in a private datacenter and hosts part of our build services. We also run our CI systems in this private datacenter. Azure is great, but also super expensive for some workloads. Hosting this workload ourselves reduces the operational cost for build services with a factor 6. This infrastructure is all SSD based, and we’re getting loads more horsepower out of the VM’s running here. To give an example: the Visual Studio 2015 Update 3 install ran for 4 hours on our Azure build agents, the private VM’s did the update in half an hour.

Number of packages and downloads

Our shared tenant currently hosts over 1 million unique packages. These packages have been downloaded 17,305,790,397 times so far. Yes, that’s over 17 billion. We handle roughly 400 million downloads each month. Some of our Enterprise customers are pushing similar numbers on their instances.

If we look at package types, roughly 50% of packages are NuGet and symbols packages, including Chocolatey and Octopus Deploy. 30% of packages are npm, with Bower and VSIX following at around 10% each.

Storage, traffic and bandwidth

Right now, our shared MyGet tenant consumes 3.2 TB of storage for just packages and metadata, including backups in secondary Azure regions.

Each month, users of www.myget.org push 2.4 TB of data into our system. Many feeds contain CI packages and retain only the latest 100 nightly builds. Other feeds contain 100 MB of Octopus Deploy artifacts, being updated daily. On the consumption side, we have 8 TB of traffic being consumed each month, spread across our Azure bill and the Switzerland-based KeyCDN.

We used to run off the Azure CDN, but found that KeyCDN has been more reliable at half the price. We still have the Azure CDN configured so we can fail-over in case it’s needed. So far, this hasn’t been required. Swiss efficiency #ftw!

Builds

When we first introduced build services, we envisioned making building NuGet (and later npm) packages as easy as possible. We have a number of open source projects running their builds on MyGet, as well as a number of companies enjoying the convention-based builds.

MyGet Build Services has produced over 65,000 successful builds (producing packages) so far. On average, these builds take 2 minutes to complete, with some people maxing out the 30 minute build time we allow.

Support requests

Ask any MyGet user and they will confirm we offer an excellent and speedy support. We typically manage to respond within the hour, unless we’re asleep. Which happens for a good 7 hours each day when night falls over the CET time zone. Our fastest response  so far providing a solution to the customer was 38 seconds.

We handle a good 45 support cases each day. Some of those are simple: we can often respond with a link to documentation. Others are more complicated and challenging. And if we can, we submit a PR to your project to resolve any code or configuration issues.

The busiest day of the week? Thursday. By far. It seems that is the day where everybody realizes the week is over halfway and things must be shipped. Unexpected to us, holidays are also quite busy. It seems many developers sneak in a few hours of coding time on those days and often want us to be part of that.

If we look at technologies, 80% of questions or client configuration problems are related to NuGet. 9% of requests are related to npm. And less than 1% for Bower and VSIX. What with the other 10%? These are typically orientation questions to see if MyGet is a good fit, or if MyGet fits in the customer’s architecture or development practice.

When we started this adventure, we never thought these numbers would become true. Thanks for making this happen and being a big part of this!

Happy packaging!

Setting an expiration time for your MyGet access tokens

From a security perspective, it is always good to have secrets that are only valid for a given amount of time. This ensures that these secrets have to be rolled over more often, resulting in a better overall security policy. Today, MyGet introduces expiring access tokens and API keys to accommodate this workflow.

From your profile page, you can manage your access tokens. The list of access tokens will always contain a primary key, and additional access tokens can be created.

Manage MyGet API keys

When creating (or editing) an access token, we can set a description to identify where the access token is being used. We can now also (optionally) set an expiration time after which the token can no longer be used. This can be done for additional tokens, as well as for the primary access token.

Create MyGet access key for accessing NuGet server

This change is live on all MyGet plans, so go ahead and set the expiration time for your access tokens!

Happy packaging!

Using build services to create Chocolatey packages

Chocolatey is a Machine Package Manager, somewhat like apt-get, built with Windows in mind. It lets us install software onto our machine, supports updates and dependencies, much like NuGet or npm do. MyGet has always supported feeds containing Chocolatey packages, making it easy to distribute software packages across teams or with customers. In this post, we’ll show you a trick on how to build Chocolatey packages using MyGet build services. It’s the least we can do as a Belgian company – our country is known for chocolates after all…

MyGet Build Services has a convention-type build approach that will create NuGet, npm and VSIX packages whenever required files or project types are available. By adding a build.cmd or build.ps1 file, this convention can be overridden – just the thing we want to do to create Chocolatey packages.

Using a little bit of PowerShell, we can call into Chocolatey’s choco.exe which handles packaging and verification. The following can be copy/pasted in a build.ps1 file in the root of a GitHub, BitBucket or VSTS repository:

Write-Host "Building Chocolatey packages..." $nuspecs = Get-ChildItem -Path $PSScriptRoot -Filter *.nuspec -Recurse foreach ($nuspec in $nuspecs) { choco pack $nuspec.FullName } $artifactsFolder = "./artifacts" Remove-Item -Path $artifactsFolder -Force -Recurse -ErrorAction SilentlyContinue New-Item $artifactsFolder -Force -Type Directory | Out-Null Move-Item *.nupkg $artifactsFolder Write-Host "Finished building Chocolatey packages."

Once a build is triggered on MyGet, this script will execute and create (and upload) Chocolatey packages to our MyGet feed, which we can then install on our system.

Happy packaging!

Using service messages to explicitly add a package to MyGet

MyGet build services is a convention-based build system that converts source code into NuGet, Npm and Vsix packages. It will compile code, run tests and collect the packages that were created and add them to your MyGet feed. Sometimes, for example when using custom build scripts or using gulp or grunt to run the build, we can’t always detect which packages were created. To add these packages to your feed, you can use service messages.

By writing a service message (a specially formatted string) to the build output, you can influence part of the build process. For example fail the build, update the version number, setting environment variables and so on. We recently added the publishPackage service message, which lets you specify additional packages to add to your feed when the build succeeds. The following example pushes the mypackage.zip to your feed as a Bower package:

##myget[publishPackage path='mypackage.zip' type='bower']

Check our documentation for additional options and remarks and give it a try. We’d love to hear your thoughts!

Happy packaging!

MyGet 2.1 Release Notes

MyGet 2.1 was released on February 12, 2016.

Highlights

This 2.1 release of MyGet again adds some new functionality to the service. Major highlights of this release are:

This means, from now on, you can now also use MyGet to host your symbols, publicly or secured.

MyGet 2.1 Highlights

Features

MyGet (all plans)

The following applies to all MyGet plans:

MyGet (paid plans)

Obviously all paid plans also get the enhancements made available on the free plan. The following applies to the MyGet Starter and Professional plans:

  • Security: Feed privilege claim tokens now expire after 15 days when unused
  • Billing: we now allow for yearly billing to reduce the administrative overhead of billing cycles

MyGet Enterprise

The enterprise plan has all functionality from the paid subscription plans, and more! The following applies only to the MyGet Enterprise plan:

  • Support for restricting feed creation to administrators only

MyGet Build Services

  • Support for explicitly publishing artifacts using service messages
  • Support for BitBucket API v2.0 and switch to OAuth 2.0
  • Support for xUnit v2.0
  • Support GitLab POST hooks
  • Performance improvements in build hook payload parsers
  • Web Hooks: new Build Queued web hook
  • New option: toggle access to build log for feed consumers

Bug Fixes

  • Various performance improvements in the web site
  • Bower: license analysis now also checks Bower package licenses
  • Usernames on feed security page now link to the user profile page
  • npm: proxy dist-tags from upstream when available
  • Fixed a bug where in some cases uploading a ZIP file Bower package threw an exception
  • Fixed a bug where in some cases pushing an npm package upstream to npmjs.com failed
  • Fixed a bug where editing a package source was not possible when the package source is unavailable

We love hearing from you, so keep that feedback coming! MyGet is built for you!

Happy packaging!

Two of my packages are treated as one. Help!

Every once in a while, our support gets a question that is similar to the following:

When requesting the package details for some versions of the NuGet packages in our feed it seems like the service encounters an error preparing the ATOM document, also resulting in NuGet failing to download the package.

Here's an example URL: https://www.myget.org/F/chucknorris/api/v2/Packages(Id='MyPackage',Version='1.2.20160209.1-master')

Is something wrong with MyGet? Definitely not! Let’s look at the packages for this feed:

  • MyPackage 1.2.20160209.1-master
  • MyPackage 1.2.20160209.1-dev

Can you spot the error? It’s very subtle, so let us help here. The version number has three dots (“.”) in it, which means it should be treated as a major.minor.patch.build type of version number, which is numeric. MyGet takes the hint and will strip off the –master and –dev in the above versions, treating both packages as one.

Clearly, the intention here was to use Semantic Versioning (SemVer): the –master and –dev suffixes to the version number were intended to provide a label in the version number. But semantic versioning is in the form of major.minor.patch-label, which means one dot less in the version number should be used. So in this case, the packages should be re-versioned and re-uploaded with the correct version number.

Don’t you guys have validation for this? is one of the questions we’d expect. We do, but it’s opt-in because many of our users start out with non-SemVer packages on their feeds and we don’t want to block those from being uploaded either. From a feed’s Package Settings tab, enable the Forbid packages which are non-compliant with Semantic Version? option.

Forbid packages which are non-compliant with Semantic Version?

This will prevent malformed semantic versions from being uploaded to your feed and prevent the above error from ever happening to you.

Happy packaging!