How to I programmatically get a list of users in a crowd directory

[Terry Harple]

I have a single Crowd Internal Directory and a Delegated authentication directory.
I have 88 users but my license count is 105.

I need to extract and compare the user properties in the crowd internal directory to the user properties in my LDAP directory in order to get my license count equal to my user count.

Somehow I feel like I am debugging Atlassian Applications but either way I need to add more users and crowd won't let you log in to fix the user count once the user count is over the license number according to their counting method. To get past this I am running on an evaluation license, and it is going to expire soon.

So I want to query the Crowd Internal directory extracting all of the pertinent user info so that I can compare the case, spacing etc. with my LDAP AD values.

All of the documentation seems to be geared for adding a new application.

Does anyone have any idea how I should go about this?

[Marcin Kempa]

Hi @Terry Harple,

I believe that you are running into this issue https://jira.atlassian.com/browse/CWD-5025. We will be shipping a fix for that in Crowd 3.1.x soon (in the following weeks).

In the mean time you can query your database in order to extract your users.

To get list of all users that Crowd has in the database you may run following query:

select 
 u.user_name, 
 u.directory_id, 
 d.directory_name, 
 d.directory_type 
from 
 cwd_user u, cwd_directory d
where 
 u.directory_id = d.id
order 
 by LOWER(u.user_name)

Note 1 in general this is not the ultimate query to get list of all users that can authenticate with Crowd. There might be a setup with a connector directory which is not cached locally in Crowd. In such situations it won't be possible to query DB to get those users. However, in this particular case described in this question we have only two directories Internal one and the delegated one, so this query will work

Note 2 this query was tested on Postgres 9.6 database

This query output will help you to figure out which users might be counted twice as their user_name case won't match.

Internally Crowd uses user_name column to determine unique users that consumes license seats (those users needs to be able to authenticate to applications defined in Crowd in order to consume license).

Once you have this list you may rename your users in the internal directory changing their usernames to what you have in your LDAP AD. If you are going to do rename in the internal directory, I suggest doing that through the UI rather than directly in the database.

Changing username case, won't have an impact on your users logging in, as Crowd login is case insensitive. In general in terms of users names, Crowd is case insensitive, case preserving. The only exception to this rule was license counting which is definitely a bug.

I hope that helps and you will be able to resolve the issue for now, before we provide an ultimate fix.

Marcin Kemp

[Marcin Kempa]

For those who are looking for getting list of all users that counts towards the Crowd license, the Crowd team has just released a new version of Crowd 3.1.3 which provide such a list in the log files.

Please follow the Crowd documentation or the KB article.

Additionally in Crowd 3.1.3 the following bug in license counting was fixed: CWD-5025

Best Regards,

Marcin Kempa

Crowd Dev Lead

[Terry Harple]

This is what I did to compare user attributes between the Internal crowd directory and the Delegated Authentication Directory.

crowd-> users -> internal directory -> search -> copy and paste all entries into a file.

crowd-> users -> Delegated Authentication Directory -> search -> copy and paste all entries into a 2nd file.

Run emacs ediff on files.

Identify fields with differing case between file entries.

Make changes in ONLY in crowd Internal directory to reflect entrees in Delegated Authentication Directory.

At 1st I changed fields in either directory trying to make them consistent, but the Delegated Authentication Directory entries must periodically resync the data because my changes were overwritten.
License count is now correct.

[Marcin Kempa]

Hi @Terry Harple,

I am glad that you were able to resolve this issue, and I am sorry that this issue caused so much problem for you. As I said before we are working on releasing new version of Crowd with a proper fix for this problem (https://jira.atlassian.com/browse/CWD-5025). It should be ready in the up coming weeks.

Best Regards,

Marcin Kempa

[Terry Harple]

Marcin

I noticed that CWD-5025 will still have a few username comparison issues. I would recommend this post to the crowd developers: https://www.b-list.org/weblog/2018/feb/11/usernames/

[Marcin Kempa]

Hi @Terry Harple,

Thank you for this link, it was an interesting and informative reading. I will definitely pass this to the Crowd development team.

Regarding the CWD-5025, I am not sure which problem you are referring to (I assume that one of those mentioned in the article).

When Crowd counts users it does that by comparing usernames. In this example (CWD-5025) there was a bug as comparison, used in license counting, did not take into account case. In other places, such as user creation and authentication, Crowd is case insensitive but case preserving. So for example it is not possible to create user 'John' and 'john' in the same directory but on the other hand when authenticating both forms 'john' and 'John' will represent the same user.

Internally username is stored as a String in the UTF-16 format in which supplementary characters are represented by surrogate pairs. Also usernames stored in remote directories such as LDAP in a 'cn' attribute (which is the default mapping for delegated dir in Crowd), according to RFC-4519 are UTF-8 so it should not be a problem with representing names coming from remote in Crowd and with making comparisons.

It is possible to create a user such as '🐙' in Crowd and if this username is present in multiple directories connected to applications in Crowd, they will be counted as one (assuming that this user can login to those applications).

I would really appreciate your feedback and if you recognized another problem in the license counting in Crowd we would love to hear about it so we can fix it before Crowd is released.

Best Regards,

Marcin Kempa

[Marcin Kempa]

Hi @Terry Harple,

Additionally to the CWD-5025, we've just released an experimental (Atlassian Labs) Crowd plugin specifically for delegated directory user pruning.

This plugin allows to optimize your license usage by periodically cleaning inactive users from your delegated directories. After configuring pruning for a delegated directory the plugin will periodically check if the directory contains any users who have been deactivated or removed from the remote directory.

By default such users will be deactivated in Crowd. It is also possible to enable "hard delete" mode, in which users who have been deleted in the remote directory will also be deleted in Crowd.

Plugin can be downloaded from Atlassian Marketplace here.

I hope that this helps you with your delegated directory configuration.

Marcin Kempa

[Bruno Vincent]

Hi @Terry Harple,

I guess you have three different options here.

First option: Database query. @Marcin Kempa detailed this very well. Since you are running on an evaluation license, maybe you're using an embedded HSQLDB database. In that case, you can run @Marcin Kempa's query in HQSQL Database Manager UI:

$ cd <crowd.installation.directory>
$ java -cp ./apache-tomcat/lib/hsqldb-1.8.0.10.jar org.hsqldb.util.DatabaseManager --url "jdbc:hsqldb:<crowd.home>/database/defaultdb"

For example:

$ cd /opt/crowd
$ java -cp ./apache-tomcat/lib/hsqldb-1.8.0.10.jar org.hsqldb.util.DatabaseManager --url "jdbc:hsqldb:/var/crowd-home/database/defaultdb"

Second option: Backup file. Unless you made some changes in Crowd's administration console > settings > Backup, Crowd does a daily backup at 2AM in <crowd.home>/shared/backups/ (Crowd v3.x) or <crowd.home>/backups/ (Crowd v2.x)

You will find all your internal directory and delegated directory users in that backup.

Third option: Crowd REST API.

I suppose you have connected one or more applications to your directories in Crowd's administration console. From the servers on which these applications are running, you can run the following command to fetch all your users.

$ curl -X GET -u 'application_name:application_password' '<crowd.server.url>/rest/usermanagement/1/search?entity-type=user'

For example:

curl -X GET -u 'jira:password' 'http://localhost:8095/crowd/rest/usermanagement/1/search?entity-type=user'

Hope this helps.

Bruno

[Terry Harple]

We have separated IT Administration from user administration so I need to work on getting local access to the servers.

In the meantime when I list users for my applications, Jira and Confluence, I have users listed that are no longer listed in either of my directories when I look via Crowd-> Users? 
In crowd-> Applications-> users, they are listed as being in my deligated authentication directory. I recently changed the user filter for the deligated directory to only include active domain users.  And I cannot delete or inactivate them in the application-> confluence -> User list. Like I said these users are not in either directories user list. Weird. I guess the update will address this?

[Bruno Vincent]

As far as I understand you don't see some of your users in the main Users tab but you do see them in Applications > your app > users. That may sound like a silly question but you do select your delegated directory in the search form of the main Users tab, don't you?

[Terry Harple]

I misspoke and was confused.  I have been using ldapsearch to refine my user query filter and I made the change to the Delegation Directories Configuration tab. But the filter does not seem to have the same effect as it did using ldapsearch on the same AD server.
Users removed using the filter are not removed from the Delegated authentication directory or the applications. The filter I am using: (&(objectCategory=person)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2))) as found on line and tested using ldapsearch.

Sorry to muddled this issue.

[Bruno Vincent]

No problem Terry, that's what the community is for!

What you get is absolutely normal. A delegated authentication directory is basically an internal directory, with a specific behaviour during user authentication since it connects to an LDAP server to check the user's credentials. So changing the filter won't remove any user in your delegated directory, it will only change the LDAP request used for authentication. What you were expecting is the standard behaviour for the LDAP/AD directory connector (which is different from the delegated authentication directory).

[Terry Harple]

Thanks again for helping with my confusion. So how do I flush out, re-sync or reset the delegated authentication directory entries? They still show up as active users in my delegated authentication directory as well as all of my applications? Hmmm. Very confusing. Not sure how to manage users I cannot delete?

[Marcin Kempa]

Hi @Terry Harple,

Indeed as @Bruno Vincent described the way delegated directories work is slightly different from connector types and those types of directories does not synch users from remote directory regularly, just on user login thus if user gets disabled or removed in the remote the user will not be removed.

In a delegated directory, you can manually remove a user using the UI. This will only remove user in the Crowd's database it will not trigger remove in the remote.

Please note that if you remove a user from a directory, the changes will be eventually synched to products connected to Crowd, which are using this directory. In Jira this would mark the user as disabled and they issues and comments will still be assigned to such user. In Confluence pages and comments made by such users will be marked as made by Unknown user (username). In BBS the name of the user should still be present. Please see following comments regarding the state in Jira, Confluence and Bitbucket Server after user removal.

Hope that helps,

Marcin Kempa

[Terry Harple]

I think we may be getting to the root of my confusion. When I attempt to delete this user in the UI I get this "Directory does not allow user removal" which could be because I have all the permission clear in the permission tab of the directory. I did this because I thought the permissions applied to the actual LDAP AD directory server. Your answer implies that the permissions apply to the internal crowd version of the LDAP AD directory. I.E. not the actual LDAP directory that all of the other fields in the Details, Connector, and Configuration tab reference? We can not have crowd administrator deleting or changing users attributes in our LDAP AD server so I cleared all of the permissions fields. Is my new understanding correct? I have another application that syncs with our LDAP server and it explicitly states that it accesses the directory read-only. Is that also the case here?

[Bruno Vincent]

A delegated authentication directory connector won't modify or delete anything in your AD directory as it is just used for authentication. The permissions you see in the permissions tab actually mean that the Crowd administrator is allowed/disallowed to modify the user data in Crowd's database.

[Terry Harple]

Disco!
Not sure that this will fix all of my issues but it certainly clears up my understanding of the relationship between Delegated Authentication Directory and the actual underlying AD LDAP server.
Thanks!

[Marcin Kempa]

In addition to what @Bruno Vincent said, those permissions are also applied when a change to the directory is triggered from application connected to Crowd which is using this directory.