MyGet Blog

Package management made easier!

NAVIGATION - SEARCH

Working with MyGet upstream sources

Upstream sources play a key role in a professional approach towards Package Management. MyGet gives you the option to specify one or more upstream sources for a package feed. Even though this feature has been available on MyGet for years now, we feel it upstream sources deserve a place in the spotlight once: they enable various scenarios that are impossible on any other package management service, and above all: they are a huge facilitator for a smooth and automated development workflow (and thus developer happiness :-)).

In this post:

  • Why upstream sources?
  • Supported upstream sources

Why use upstream sources?

In bullet-form:

  • Upstream sources make it very easy to pull in packages from other package sources onto your downstream MyGet feeds.
  • You can also target these upstream sources to push packages upstream from your MyGet feeds.
  • Any configured upstream source on your MyGet feed will be made available to you in MyGet Build Services, without having to commit any secrets such as credentials or API keys in your source repository. If you're using NuGet: no need to create a nuget.config file!

If you are confused about the usage of "upstream" and "downstream" in the context of package sources, we've got a little poetic explanation for you, which may already help you visualize the relationship between package consumers, your MyGet feeds, and your feed's upstream sources.

Consider the direction in which packages are flowing from a given package source to an ocean of consumers.Your package may have dependencies "upstream", to packages on another feed. From the point of view of those dependencies, the depending package is located "downstream". When a user consumes the downstream package, it will also fetch the upstream dependencies.The consumer, however, is only allowed to fetch or query those upstream packages if the feed being queried (downstream) is also configured to proxy (or mirror) the upstream package source.


Supported upstream sources

By default, MyGet feeds have the public, central repositories configured for each package type. This includes:

  • NuGet: https://www.nuget.org/api/v2
  • Bower: https://bower.herokuapp.com
  • npm: http://registry.npmjs.org
  • Maven: https://repo1.maven.org/maven2

To configure an additional upstream source for your MyGet feed, navigate to Feed Settings > Upstream Sources. Then click Add Upstream Source and select the package source type you want to add.

A dialog will prompt your for upstream source information and will also expose a few common presets for you to take advantage of. Did you know we even support Dropbox?!

Upstream Source Credentials

If you have any access privileges to other MyGet feeds, you will see those in the MyGet Feeds presets, so you can easily build a chain of package sources to facilitate a package promotion flow.

If you select a private MyGet feed you have access to as an upstream source, there's no need to provide credentials to be able to restore packages from it on MyGet Build Services. MyGet will impersonate your user account when authenticating against that upstream source.

For any non-MyGet upstream source that requires authentication to pull packages, you'll have to provide username and password to be used during Basic Authentication.

Warning! Be very careful with password managers and browser add-ons providing auto-completion of credentials!
We recommend disabling these credential managers on the MyGet web site to avoid issues when editing upstream sources. Oftentimes, these tools auto-complete the credentials fields with out-dated credentials (even when editing different settings in the dialog). 
When running into package restore failures on MyGet Build Services, or when noticing that upstream packages are no longer available downstream, this is the most common source of the issue.

In the opposite direction, in order to push packages from your downstream MyGet feed to the upstream source, you may need to configure a (scoped) API key or access token.

Package Source Filtering

Applies to: NuGet (v2 only!)

When the upstream source is a v2 NuGet feed, you may configure additional OData filtering. Filtering is based on the OData v3 Filtering System. Valid filters are similar to Id eq 'jQuery' or IsLatestVersion eq true and Id ne 'Foo'.

Warning! This capability may go away at some point in favor of newer NuGet v3 APIs.
We currently still keep the feature around for some scenarios that are not yet fully supported on NuGet v3.

Adding a package from an upstream source

You can easily add packages to your MyGet feed originating from an upstream source, such as nuget.org, nmpjs.org, etc. This is using the feed's configured upstream sources under the hood. If you want to add a package from another feed onto your MyGet feed, the other feed needs to be configured as an upstream source to that feed.

Adding a package from an upstream source can happen in three ways: manually, by reference (proxying), or by value (mirroring).

  • Manually: you can add packages from an upstream source to your feed manually by using the Add Package button you will find under your feed's Packages page.

Select From Feed in the dialog that prompts.

  • Proxying: the package metadata is copied to the MyGet feed, the package itself remains hosted on the upstream source. When querying the package, we call the upstream source to fetch the package.
  • Mirroring: the package metadata and the package itself are copied onto the MyGet feed. When querying the package, we server the package directly and don't use the upstream source. Mirroring of a package version happens upon the first request for that given package version.

Configuring upstream sources on your MyGet feed unlocks quite a few integration scenarios and automation opportunities!

Proxy packages from an upstream source

You can configure an upstream source to proxy upstream packages through your MyGet feed to your feed consumers. Proxying makes it easy to have a single MyGet feed aggregate packages from multiple sources. Package consumers need only to configure a single MyGet feed, and all packages available on upstream, proxied package sources will become available to them.

Benefits:

  • upstream packages do not count against your MyGet storage quota
  • authentication against upstream, private MyGet sources happens automatically (see Upstream Source Credentials)
  • especially for NuGet: no longer subject to chatty clients reaching out to all configured feeds during dependency resolution (MyGet can be smarter server-side)

Drawbacks:

  • every package request will incur additional latency as opposed to storing the package onto the MyGet feed

Warning! Avoid configuring multiple package source proxies on a single feed, or in a chain of feeds, as this will magnify the disadvantages, and result in very slow feed response times.

The following diagram illustrates the effects of upstream source proxying.

To enable upstream source proxying, you must tick the check-mark next to Make all upstream packages available in clients.

Mirror packages from an upstream source

You can configure an upstream source to mirror upstream packages onto your MyGet feed. This configuration is similar to package proxying, but takes it one step further.

Whenever someone requests a particular package from your MyGet feed for the first time, MyGet will query the upstream source and copy the package onto the MyGet feed.

Benefits:

  • No additional latency (except for the first hit that triggers the package mirroring)
  • Protected against upstream source availability issues
  • Protected against upstream package removal
  • Authentication against upstream, private MyGet feeds happens automatically (see Upstream Source Credentials)
  • Faster builds!

Drawbacks:

  • Mirrored packages count towards your MyGet storage quota (a classic storage versus speed trade-off, you can always upgrade your subscription or request a quote!)

The following diagram illustrates the effects of upstream source mirroring.

To enable upstream source mirroring, you must tick the check-mark next to Automatically add downloaded upstream packages to the current feed (mirror).

Optionally, you can also tick the third check-mark to indicate that any package found upstream is to be considered a package dependency (and should not be consumed directly). This will hide those packages from search results, whilst still allowing you to restore them.

Once upstream source mirroring is enabled, we can consume our MyGet feed from Visual Studio which will also list upstream packages. For example, the example acmecompany feed only lists one package:

One package on our feed

When searching in Visual Studio, we do see packages that originate from upstream sources:

Visual Studio showing upstream packages

After installing this package, our feed now automatically contains a copy of the jQuery package:

Mirror upstream pckages

From now on, the package is available from our MyGet feed directly, without having to explicitly add it manually from the upstream source.

Using a MyGet feed as a staging area (before pushing upstream)

Many development teams are using some kind of package promotion workflow: pushing a package from one feed to another based on quality gates, target audience, or any other criteria. This is very typical scenario for which upstream sources are essential.

Of course, all of this can happen in an automated fashion using package manager client. However, as promoting a package typically involves some kind of human intervention (e.g. release manager approval), we've also made it a first-class feature in the MyGet web site.

Simply pick the package version you want to promote from the package details page, and hit the Push button to initiate the package promotion flow.

A dialog will provide you with additional options. MyGet is also smart enough to detect any package dependencies you might want to push along in one go as part of this package promotion flow.

At this point, you can still make a few metadata changes before pushing upstream. This dialog allows you to:

  • modify or remove the prerelease label of the upstream package version. This allows you to e.g. drop the prerelease label to release a package without rebuilding/repackaging.
  • add release notes to be included in the package metadata. MyGet will even support release notes written in markdown and render them properly on the web site!
  • modify or remove the SemVer2 build metadata part of the upstream package version
  • exclude any detected dependencies or satellite packages from the push action
  • apply source labeling if the package was built using MyGet Build Services. When enabled, MyGet will find the build from which the package originated and will add a label to the source control revision it was built from.

To edit a package's metadata, simply click the Edit button next to it and make the modifications. To apply a given modification to all packages in the dialog, hit the rain drop button next to the editable field.

Using upstream sources on MyGet Build Services

Applies to: NuGet, npm

Upstream sources for a feed are also available during build. This can be useful in the following scenarios:

  • An additional upstream source is needed during build. MyGet will make the upstream source available during build if it has been added to the feed's upstream sources.
  • If you have a private feed requiring authentication but do not wish to add credentials to source control, credentials can be added to the feed's upstream source. These credentials will be available during build and allow you to consume a protected feed with ease.

Applies to: NuGet

  • The API key for an upstream source is also made available during the build process. This means during a build, you can call into nuget.exe push and push packages to configured upstream sources.
  • If you want to make use of nuget.exe push in a build script without having to specify the -Source parameter. This requires a default upstream source to be defined.

Applies to: npm
We strongly suggest to proxy registry.npmjs.org to be able to run `npm install` during build, as npm will default to the MyGet feed as the default registry.

Setting default upstream sources to be used on MyGet Build Services

Applies to: NuGet

The NuGet.config file on our build agents is configured using NuGet's defaults, enriched with all NuGet upstream sources configured for your MyGet feed. Based on these defaults, the following conventions are active:

  • The default upstream source is set to (Aggregate Source), meaning all feeds will be queried for packages in the order defined in the feed's upstream sources.
  • The default push source (when using nuget push without the -Source parameter) is NuGet.org.

Both of these conventions can be overridden by editing the build source configuration.

Auto updating packages

MyGet feeds can automatically fetch package updates made available through the upstream sources.

When adding or editing a upstream source, we can enable this behaviour per package source, as well as an interval when MyGet should check for updates.

Package Source Options

The following options are available:

  • E-mail me when package updates are available: Sends an e-mail to the specified recipient(s) when package updates are available on the upstream source.
  • Include prerelease versions: By default, MyGet will only consider stable packages. When enabled, we will also check pre-release packages from the upstream source.
  • Automatically update packages to their latest versions: Enables the behavior of automatically updating packages from the upstream source.
  • Update interval: Depending on your subscription plan, we can specify how often MyGet should check for updates (up to every 30 minutes on a Professional subscription)

As you can see, MyGet's support for upstream package sources unlocks a wide range of package management scenarios that may help you streamline your development flow and package governance even more. If you haven't tried the above scenarios yet, do give them a try and experience how it may make your life easier.

Oh, and we do support pushing your private symbols packages upstream along with your NuGet packages, too!

Happy Packaging!

PS: Please take 10 seconds of your precious time to tell us how we're doing

Configure which feed a token can push packages to - introducing feed-scoped access tokens

Many development teams are making use of a continuous integration server like TeamCity, Jenkins or VSTS to build their projects and push generated NuGet, npm, Bower and VSIX packages to their MyGet feed. When having multiple feeds, it is a good practice to limit the feeds this access token/API key can push packages to, ensuring the surface area of the specific access token is limited to just the feeds the access token requires access to.

In short, scoped access tokens:

  • Are a good security best-practice: use minimum required permissions for a specific operation
  • Avoid services/users accidentally pushing packages by using read-only tokens where possible
  • Allow pushing packages without the ability to get access to other packages on the feed (write-only)

New access tokens and existing access tokens can be scoped in terms of what they can do. We now let you to create read-only or write-only access tokens, optionally limiting write access to just one specific feed.

Create new access token scoped to a given feed

Next to scopes, the access token expiration date and time can also be specified, making it possible to create a time-limited access token that has to be recreated to continue having access to the feed.

Happy packaging!

Checking potential vulnerabilities in project dependencies

Software projects nowadays are based on many third party and open source libraries. It is important to be aware of any potential security vulnerabilities in these components, to ensure our own software project is secure. Thanks to OSSIndex and Vor Security, we now have a vulnerability report ready for your MyGet feed!

While still in preview, every feed now has a Vulnerabilities tab which reports potential vulnerabilities in packages on that feed, whether NuGet, npm or Bower.

vulnerability-report

The vulnerability report provides us with an overview of potential vulnerabilities in our dependencies. We can also see the percentage of packages with potential vulnerabilities versus the percentage of packages with no known vulnerabilities.

Give it a go, we’re looking forward to your feedback on this new feature! Leave your comments below or reach out on Twitter.

Happy packaging!

Keeping feeds clean with retention rules

MyGet Package Retention Rules help clean up your NuGet npm feedMany developer teams use MyGet for storing their continuous integration and/or nightly builds of NuGet, npm, Bower and VSIX packages. As more and more packages get added, it may become harder to manage them all. Some packages may be used in projects, while others are not. Let’s go over the options available for housekeeping.

By default, MyGet keeps all package versions available on our feeds. Every package pushed is there forever, unless manually removed or removed by package retention. By setting retention rules, it is possible to automatically trim the list of packages to X latest packages, keeping into account package usage in projects and package dependency trees.

Configuring retention rules

Retention rules are defined per feed. Some feeds may have more aggressive retention rules defined, other may not have them enabled at all. From the Retention Rules, we can define:

  • the maximum number of stable versions to keep
  • the maximum number of prerelease versions to keep
  • whether to keep depended packages or not – enabling this makes sure package restores always complete successfully by keeping the dependency tree in its entirety
  • whether to allow removal of packages that have downloads – enabling this option ensures that packages that are being used in projects never get deleted

Setting retention rules

Keeping a specific package around

Retention rules are quite brute-force: they will always remove all packages that match the configured rules. Luckily, MyGet lets us “pin” packages which we want to keep around. For example, we may want to only keep the latest 10 pre-release versions while still keeping around the 20th pre-release version we’re still using in our projects.

From the package details page, we can define which package versions should never be considered by retention rules by using the Pin button next to the package.

Pinning packages so they do not get removed

We can pin package per version, or all versions at once using the button at the top of the Package History list. Of course, we can also Unpin packages using the same approach. Once a package is unpinned, retention rules are allowed to remove them.

Custom retention rules using web hooks

Using the built-in retention rules may not be enough. For example, what if we want to run retention rules based on other conditions than the latest version? What if we want to only remove packages when there is a full moon? Using web hooks, we can subscribe to certain feed events (like “package added”) and run our custom logic to optionally remove packages from our feed. We have a complete example available that helps getting started with web hooks.

Learn more about package retention in our documentation.

Happy packaging!

Dropbox as a package source for NuGet, npm, Bower and VSIX packages

Wouldn’t it be awesome if creating a NuGet, npm, Bower or VSIX feed was as easy as just copying packages into a Dropbox folder? Awesomeness is here: we’ve added Dropbox as a package source type to MyGet. This allows us to link a Dropbox folder to a MyGet feed and automatically upload packages so they can be consumed with the popular package managers out there.

Synchronizing NuGet packages with Dropbox

The Dropbox package source makes it easy to move packages into MyGet. For example, migrating from a network share to MyGet is as easy as copy-paste. Have a build server that drops artifacts into a Dropbox folder? MyGet will add the synchronized artifacts to your feed. Right now we download packages from Dropbox on a schedule (every 15 minutes).

Give it a try and let us know how it goes – feedback is welcome through the comments below or via the MyGetTeam Twitter account.

Happy packaging!

Package details showing GitHub project README

We’re happy to introduce a few user interface enhancements which have been available for all users of npm private feeds on MyGet. We’ve now rolled these out to the package details page for NuGet, npm, Bower and Vsix packages. These pages now display all “at-a-glance” information on the right. Package owners, authors, license information and downloads can be seen from here.

The wider part of the package details page now displays the README.md contents retrieved from GitHub. That is, if the GitHub repository is accessible for us. This makes it easier for consumers of your feed to see installation instructions, links to documentation and so on from the package details page.

MyGet showing GitHub readme contents on package details page

We're looking forward to hearing your feedback through the comments below. Or tweet us via @MyGetTeam.

Happy packaging!

MyGet now offers NuGet, Npm and Bower registries

pmsWith our latest MyGet release, we’ve added support for npm and bower registries. We’ve always been very focused on building a great story around NuGet and decided it was time for Npm and Bower enthusiasts to get a similar experience.

Adding npm and Bower support was high on our wish list and that of our users. Many developers are doing only front-end development and need a public or private npm registry. Others are working with DNX (the new name for ASP.NET 5 or ASP.NET vNext) and combine NuGet, npm and Bower. It feels good to be able to support them all!

To help you get started, we’ve prepared a few short tutorials that help you get started on MyGet with these package managers:

Oh and build services now also packages node modules! Just point MyGet to your GitHub repository and we’ll package your npm packages, too.

We really look forward to hearing your feedback on this!

Happy packaging!